Sep 17 17:38:47.995953 ip-10-0-139-237 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Sep 17 17:38:47.995963 ip-10-0-139-237 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Sep 17 17:38:47.995970 ip-10-0-139-237 systemd[1]: kubelet.service: Failed with result 'resources'.
Sep 17 17:38:47.996113 ip-10-0-139-237 systemd[1]: Failed to start Kubernetes Kubelet.
Sep 17 17:38:58.134453 ip-10-0-139-237 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Sep 17 17:38:58.134470 ip-10-0-139-237 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot ada3a6c519584767997d7e6651745c9d --
Sep 17 17:40:57.329884 ip-10-0-139-237 systemd[1]: Starting Kubernetes Kubelet...
Sep 17 17:41:01.115913 ip-10-0-139-237 kubenswrapper[2567]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 17 17:41:01.115913 ip-10-0-139-237 kubenswrapper[2567]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 17 17:41:01.115913 ip-10-0-139-237 kubenswrapper[2567]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 17 17:41:01.115913 ip-10-0-139-237 kubenswrapper[2567]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 17 17:41:01.115913 ip-10-0-139-237 kubenswrapper[2567]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 17 17:41:01.116752 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:01.115988    2567 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 17 17:41:01.119334 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.119316    2567 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 17:41:01.119334 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.119333    2567 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 17:41:01.119402 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.119337    2567 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 17:41:01.119402 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.119341    2567 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 17:41:01.119402 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.119344    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 17:41:01.119402 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.119347    2567 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 17:41:01.119402 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.119350    2567 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 17:41:01.119402 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.119353    2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 17:41:01.119402 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.119356    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 17:41:01.119402 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.119359    2567 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 17:41:01.119402 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.119361    2567 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 17:41:01.119402 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.119364    2567 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 17:41:01.119402 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.119367    2567 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 17:41:01.119402 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.119370    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 17:41:01.119402 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.119372    2567 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 17:41:01.119402 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.119375    2567 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 17:41:01.119402 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.119377    2567 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 17:41:01.119402 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.119380    2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 17:41:01.119402 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.119382    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 17:41:01.119402 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.119385    2567 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 17:41:01.119402 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.119387    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 17:41:01.119402 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.119391    2567 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 17:41:01.119901 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.119398    2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 17:41:01.119901 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.119401    2567 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 17:41:01.119901 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.119403    2567 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 17:41:01.119901 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.119406    2567 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 17:41:01.119901 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.119409    2567 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 17:41:01.119901 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.119412    2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 17:41:01.119901 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.119415    2567 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 17:41:01.119901 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.119418    2567 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 17:41:01.119901 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.119420    2567 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 17:41:01.119901 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.119423    2567 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 17:41:01.119901 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.119425    2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 17:41:01.119901 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.119428    2567 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 17:41:01.119901 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.119431    2567 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 17:41:01.119901 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.119435    2567 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 17:41:01.119901 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.119440    2567 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 17:41:01.119901 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.119442    2567 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 17:41:01.119901 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.119445    2567 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 17:41:01.119901 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.119448    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 17:41:01.119901 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.119450    2567 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 17:41:01.119901 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.119453    2567 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 17:41:01.120391 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.119456    2567 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 17:41:01.120391 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.119458    2567 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 17:41:01.120391 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.119461    2567 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 17:41:01.120391 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.119463    2567 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 17:41:01.120391 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.119466    2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 17:41:01.120391 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.119469    2567 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 17:41:01.120391 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.119471    2567 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 17:41:01.120391 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.119475    2567 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 17:41:01.120391 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.119478    2567 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 17:41:01.120391 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.119480    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 17:41:01.120391 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.119483    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 17:41:01.120391 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.119485    2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 17:41:01.120391 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.119488    2567 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 17:41:01.120391 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.119494    2567 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 17:41:01.120391 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.119497    2567 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 17:41:01.120391 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.119499    2567 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 17:41:01.120391 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.119502    2567 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 17:41:01.120391 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.119506    2567 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 17:41:01.120391 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.119510    2567 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 17:41:01.120878 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.119513    2567 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 17:41:01.120878 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.119517    2567 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 17:41:01.120878 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.119519    2567 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 17:41:01.120878 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.119522    2567 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 17:41:01.120878 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.119525    2567 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 17:41:01.120878 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.119543    2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 17:41:01.120878 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.119546    2567 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 17:41:01.120878 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.119549    2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 17:41:01.120878 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.119552    2567 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 17:41:01.120878 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.119556    2567 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 17:41:01.120878 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.119563    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 17:41:01.120878 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.119567    2567 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 17:41:01.120878 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.119571    2567 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 17:41:01.120878 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.119575    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 17:41:01.120878 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.119580    2567 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 17:41:01.120878 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.119584    2567 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 17:41:01.120878 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.119587    2567 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 17:41:01.120878 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.119590    2567 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 17:41:01.120878 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.119592    2567 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 17:41:01.121342 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.119595    2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 17:41:01.121342 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.119597    2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 17:41:01.121342 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.119600    2567 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 17:41:01.121342 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.119603    2567 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 17:41:01.121342 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.119605    2567 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 17:41:01.121342 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.119608    2567 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 17:41:01.121342 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.119611    2567 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 17:41:01.121342 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.119615    2567 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 17:41:01.121342 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.119618    2567 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 17:41:01.121342 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.119620    2567 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 17:41:01.121342 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.119623    2567 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 17:41:01.121342 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.119626    2567 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 17:41:01.121342 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.119629    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 17:41:01.121342 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.119631    2567 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 17:41:01.121342 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.119634    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 17:41:01.121342 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.119636    2567 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 17:41:01.121342 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.119639    2567 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 17:41:01.121342 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.119642    2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 17:41:01.121342 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.119645    2567 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 17:41:01.121342 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.121326    2567 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 17:41:01.121870 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.121334    2567 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 17:41:01.121870 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.121337    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 17:41:01.121870 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.121341    2567 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 17:41:01.121870 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.121344    2567 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 17:41:01.121870 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.121347    2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 17:41:01.121870 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.121350    2567 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 17:41:01.121870 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.121353    2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 17:41:01.121870 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.121356    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 17:41:01.121870 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.121358    2567 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 17:41:01.121870 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.121361    2567 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 17:41:01.121870 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.121363    2567 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 17:41:01.121870 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.121366    2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 17:41:01.121870 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.121368    2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 17:41:01.121870 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.121371    2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 17:41:01.121870 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.121374    2567 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 17:41:01.121870 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.121377    2567 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 17:41:01.121870 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.121379    2567 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 17:41:01.121870 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.121382    2567 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 17:41:01.121870 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.121385    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 17:41:01.121870 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.121387    2567 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 17:41:01.122365 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.121390    2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 17:41:01.122365 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.121392    2567 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 17:41:01.122365 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.121395    2567 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 17:41:01.122365 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.121397    2567 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 17:41:01.122365 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.121400    2567 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 17:41:01.122365 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.121402    2567 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 17:41:01.122365 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.121405    2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 17:41:01.122365 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.121407    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 17:41:01.122365 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.121410    2567 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 17:41:01.122365 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.121412    2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 17:41:01.122365 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.121414    2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 17:41:01.122365 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.121417    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 17:41:01.122365 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.121419    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 17:41:01.122365 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.121424    2567 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 17:41:01.122365 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.121426    2567 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 17:41:01.122365 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.121429    2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 17:41:01.122365 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.121432    2567 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 17:41:01.122365 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.121435    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 17:41:01.122365 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.121438    2567 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 17:41:01.122365 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.121440    2567 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 17:41:01.122887 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.121443    2567 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 17:41:01.122887 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.121445    2567 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 17:41:01.122887 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.121447    2567 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 17:41:01.122887 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.121450    2567 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 17:41:01.122887 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.121452    2567 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 17:41:01.122887 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.121455    2567 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 17:41:01.122887 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.121457    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 17:41:01.122887 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.121460    2567 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 17:41:01.122887 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.121463    2567 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 17:41:01.122887 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.121465    2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 17:41:01.122887 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.121468    2567 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 17:41:01.122887 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.121470    2567 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 17:41:01.122887 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.121473    2567 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 17:41:01.122887 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.121475    2567 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 17:41:01.122887 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.121478    2567 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 17:41:01.122887 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.121480    2567 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 17:41:01.122887 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.121483    2567 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 17:41:01.122887 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.121485    2567 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 17:41:01.122887 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.121488    2567 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 17:41:01.122887 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.121490    2567 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 17:41:01.123385 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.121495    2567 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 17:41:01.123385 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.121499    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 17:41:01.123385 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.121502    2567 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 17:41:01.123385 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.121506    2567 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 17:41:01.123385 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.121510    2567 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 17:41:01.123385 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.121514    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 17:41:01.123385 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.121517    2567 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 17:41:01.123385 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.121520    2567 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 17:41:01.123385 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.121523    2567 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 17:41:01.123385 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.121525    2567 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 17:41:01.123385 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.121542    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 17:41:01.123385 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.121545    2567 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 17:41:01.123385 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.121547    2567 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 17:41:01.123385 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.121550    2567 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 17:41:01.123385 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.121552    2567 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 17:41:01.123385 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.121556    2567 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 17:41:01.123385 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.121560    2567 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 17:41:01.123385 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.121563    2567 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 17:41:01.123843 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.121565    2567 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 17:41:01.123843 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.121568    2567 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 17:41:01.123843 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.121571    2567 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 17:41:01.123843 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.121574    2567 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 17:41:01.123843 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.121577    2567 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 17:41:01.123843 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.121580    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 17:41:01.123843 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.121582    2567 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 17:41:01.123843 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.121585    2567 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 17:41:01.123843 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.121588    2567 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 17:41:01.123843 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.121590    2567 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 17:41:01.123843 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.121593    2567 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 17:41:01.123843 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.121596    2567 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 17:41:01.123843 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.121598    2567 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 17:41:01.123843 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.121601    2567 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 17:41:01.123843 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.121603    2567 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 17:41:01.123843 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.121606    2567 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 17:41:01.123843 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.121609    2567 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 17:41:01.123843 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.121612    2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 17:41:01.123843 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.121615    2567 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 17:41:01.123843 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.121618    2567 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 17:41:01.124340 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.122009    2567 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 17:41:01.124340 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.122014    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 17:41:01.124340 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.122017    2567 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 17:41:01.124340 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.122020    2567 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 17:41:01.124340 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.122023    2567 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 17:41:01.124340 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.122026    2567 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 17:41:01.124340 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.122029    2567 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 17:41:01.124340 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.122031    2567 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 17:41:01.124340 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.122034    2567 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 17:41:01.124340 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.122037    2567 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 17:41:01.124340 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.122039    2567 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 17:41:01.124340 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.122042    2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 17:41:01.124340 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.122044    2567 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 17:41:01.124340 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.122047    2567 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 17:41:01.124340 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.122049    2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 17:41:01.124340 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.122052    2567 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 17:41:01.124340 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.122054    2567 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 17:41:01.124340 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.122057    2567 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 17:41:01.124340 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.122060    2567 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 17:41:01.124340 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.122062    2567 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 17:41:01.124850 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.122065    2567 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 17:41:01.124850 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.122068    2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 17:41:01.124850 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.122070    2567 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 17:41:01.124850 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.122073    2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 17:41:01.124850 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.122075    2567 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 17:41:01.124850 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.122078    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 17:41:01.124850 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.122080    2567 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 17:41:01.124850 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.122083    2567 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 17:41:01.124850 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.122085    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 17:41:01.124850 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.122088    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 17:41:01.124850 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.122090    2567 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 17:41:01.124850 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.122094    2567 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 17:41:01.124850 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.122096    2567 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 17:41:01.124850 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.122099    2567 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 17:41:01.124850 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.122102    2567 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 17:41:01.124850 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.122105    2567 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 17:41:01.124850 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.122107    2567 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 17:41:01.124850 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.122110    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 17:41:01.124850 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.122113    2567 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 17:41:01.124850 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.122115    2567 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 17:41:01.125338 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.122119    2567 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 17:41:01.125338 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.122122    2567 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 17:41:01.125338 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.122125    2567 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 17:41:01.125338 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.122127    2567 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 17:41:01.125338 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.122130    2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 17:41:01.125338 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.122132    2567 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 17:41:01.125338 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.122135    2567 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 17:41:01.125338 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.122137    2567 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 17:41:01.125338 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.122140    2567 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 17:41:01.125338 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.122142    2567 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 17:41:01.125338 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.122145    2567 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 17:41:01.125338 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.122147    2567 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 17:41:01.125338 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.122151    2567 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 17:41:01.125338 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.122154    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 17:41:01.125338 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.122157    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 17:41:01.125338 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.122160    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 17:41:01.125338 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.122163    2567 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 17:41:01.125338 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.122165    2567 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 17:41:01.125338 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.122167    2567 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 17:41:01.125832 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.122170    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 17:41:01.125832 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.122173    2567 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 17:41:01.125832 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.122175    2567 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 17:41:01.125832 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.122178    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 17:41:01.125832 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.122190    2567 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 17:41:01.125832 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.122193    2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 17:41:01.125832 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.122196    2567 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 17:41:01.125832 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.122199    2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 17:41:01.125832 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.122202    2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 17:41:01.125832 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.122204    2567 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 17:41:01.125832 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.122207    2567 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 17:41:01.125832 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.122211    2567 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 17:41:01.125832 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.122213    2567 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 17:41:01.125832 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.122216    2567 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 17:41:01.125832 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.122218    2567 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 17:41:01.125832 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.122221    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 17:41:01.125832 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.122223    2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 17:41:01.125832 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.122226    2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 17:41:01.125832 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.122228    2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 17:41:01.125832 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.122231    2567 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 17:41:01.126329 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.122233    2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 17:41:01.126329 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.122236    2567 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 17:41:01.126329 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.122239    2567 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 17:41:01.126329 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.122243    2567 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 17:41:01.126329 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.122246    2567 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 17:41:01.126329 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.122250    2567 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 17:41:01.126329 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.122253    2567 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 17:41:01.126329 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.122256    2567 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 17:41:01.126329 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.122259    2567 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 17:41:01.126329 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.122262    2567 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 17:41:01.126329 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.122265    2567 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 17:41:01.126329 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.122268    2567 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 17:41:01.126329 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.122271    2567 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 17:41:01.126329 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.122274    2567 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 17:41:01.126329 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.122276    2567 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 17:41:01.126329 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.122279    2567 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 17:41:01.126329 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.122283    2567 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 17:41:01.126329 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.122286    2567 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 17:41:01.126329 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.122288    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 17:41:01.126840 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.122291    2567 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 17:41:01.126840 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:01.122375    2567 flags.go:64] FLAG: --address="0.0.0.0"
Sep 17 17:41:01.126840 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:01.122421    2567 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 17 17:41:01.126840 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:01.122429    2567 flags.go:64] FLAG: --anonymous-auth="true"
Sep 17 17:41:01.126840 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:01.122434    2567 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 17 17:41:01.126840 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:01.122439    2567 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 17 17:41:01.126840 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:01.122443    2567 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 17 17:41:01.126840 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:01.122447    2567 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 17 17:41:01.126840 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:01.122452    2567 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 17 17:41:01.126840 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:01.122455    2567 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 17 17:41:01.126840 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:01.122458    2567 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 17 17:41:01.126840 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:01.122462    2567 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 17 17:41:01.126840 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:01.122465    2567 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 17 17:41:01.126840 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:01.122468    2567 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 17 17:41:01.126840 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:01.122471    2567 flags.go:64] FLAG: --cgroup-root=""
Sep 17 17:41:01.126840 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:01.122476    2567 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 17 17:41:01.126840 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:01.122480    2567 flags.go:64] FLAG: --client-ca-file=""
Sep 17 17:41:01.126840 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:01.122483    2567 flags.go:64] FLAG: --cloud-provider="external"
Sep 17 17:41:01.126840 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:01.122486    2567 flags.go:64] FLAG: --cluster-dns="[]"
Sep 17 17:41:01.126840 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:01.122490    2567 flags.go:64] FLAG: --cluster-domain=""
Sep 17 17:41:01.126840 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:01.122493    2567 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 17 17:41:01.126840 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:01.122496    2567 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 17 17:41:01.126840 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:01.122499    2567 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 17 17:41:01.126840 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:01.122503    2567 flags.go:64] FLAG: --container-log-max-files="5"
Sep 17 17:41:01.127414 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:01.122507    2567 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 17 17:41:01.127414 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:01.122510    2567 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 17 17:41:01.127414 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:01.122513    2567 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 17 17:41:01.127414 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:01.122516    2567 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 17 17:41:01.127414 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:01.122519    2567 flags.go:64] FLAG: --contention-profiling="false"
Sep 17 17:41:01.127414 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:01.122524    2567 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 17 17:41:01.127414 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:01.122539    2567 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 17 17:41:01.127414 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:01.122542    2567 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 17 17:41:01.127414 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:01.122545    2567 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 17 17:41:01.127414 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:01.122549    2567 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 17 17:41:01.127414 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:01.122552    2567 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 17 17:41:01.127414 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:01.122555    2567 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 17 17:41:01.127414 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:01.122558    2567 flags.go:64] FLAG: --enable-load-reader="false"
Sep 17 17:41:01.127414 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:01.122561    2567 flags.go:64] FLAG: --enable-server="true"
Sep 17 17:41:01.127414 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:01.122567    2567 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 17 17:41:01.127414 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:01.122572    2567 flags.go:64] FLAG: --event-burst="100"
Sep 17 17:41:01.127414 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:01.122576    2567 flags.go:64] FLAG: --event-qps="50"
Sep 17 17:41:01.127414 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:01.122579    2567 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 17 17:41:01.127414 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:01.122582    2567 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 17 17:41:01.127414 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:01.122585    2567 flags.go:64] FLAG: --eviction-hard=""
Sep 17 17:41:01.127414 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:01.122589    2567 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 17 17:41:01.127414 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:01.122592    2567 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 17 17:41:01.127414 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:01.122595    2567 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 17 17:41:01.127414 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:01.122599    2567 flags.go:64] FLAG: --eviction-soft=""
Sep 17 17:41:01.127414 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:01.122602    2567 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 17 17:41:01.128086 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:01.122605    2567 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 17 17:41:01.128086 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:01.122608    2567 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 17 17:41:01.128086 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:01.122611    2567 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 17 17:41:01.128086 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:01.122614    2567 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 17 17:41:01.128086 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:01.122616    2567 flags.go:64] FLAG: --fail-swap-on="true"
Sep 17 17:41:01.128086 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:01.122619    2567 flags.go:64] FLAG: --feature-gates=""
Sep 17 17:41:01.128086 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:01.122623    2567 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 17 17:41:01.128086 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:01.122626    2567 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 17 17:41:01.128086 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:01.122630    2567 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 17 17:41:01.128086 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:01.122633    2567 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 17 17:41:01.128086 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:01.122636    2567 flags.go:64] FLAG: --healthz-port="10248"
Sep 17 17:41:01.128086 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:01.122639    2567 flags.go:64] FLAG: --help="false"
Sep 17 17:41:01.128086 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:01.122644    2567 flags.go:64] FLAG: --hostname-override="ip-10-0-139-237.ec2.internal"
Sep 17 17:41:01.128086 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:01.122647    2567 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 17 17:41:01.128086 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:01.122650    2567 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 17 17:41:01.128086 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:01.122653    2567 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 17 17:41:01.128086 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:01.122657    2567 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 17 17:41:01.128086 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:01.122661    2567 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 17 17:41:01.128086 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:01.122664    2567 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 17 17:41:01.128086 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:01.122667    2567 flags.go:64] FLAG: --image-service-endpoint=""
Sep 17 17:41:01.128086 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:01.122669    2567 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 17 17:41:01.128086 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:01.122672    2567 flags.go:64] FLAG: --kube-api-burst="100"
Sep 17 17:41:01.128086 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:01.122675    2567 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 17 17:41:01.128677 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:01.122679    2567 flags.go:64] FLAG: --kube-api-qps="50"
Sep 17 17:41:01.128677 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:01.122682    2567 flags.go:64] FLAG: --kube-reserved=""
Sep 17 17:41:01.128677 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:01.122685    2567 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 17 17:41:01.128677 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:01.122688    2567 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 17 17:41:01.128677 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:01.122691    2567 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 17 17:41:01.128677 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:01.122694    2567 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 17 17:41:01.128677 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:01.122696    2567 flags.go:64] FLAG: --lock-file=""
Sep 17 17:41:01.128677 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:01.122700    2567 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 17 17:41:01.128677 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:01.122703    2567 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 17 17:41:01.128677 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:01.122706    2567 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 17 17:41:01.128677 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:01.122712    2567 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 17 17:41:01.128677 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:01.122715    2567 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 17 17:41:01.128677 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:01.122718    2567 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 17 17:41:01.128677 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:01.122721    2567 flags.go:64] FLAG: --logging-format="text"
Sep 17 17:41:01.128677 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:01.122724    2567 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 17 17:41:01.128677 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:01.122727    2567 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 17 17:41:01.128677 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:01.122730    2567 flags.go:64] FLAG: --manifest-url=""
Sep 17 17:41:01.128677 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:01.122733    2567 flags.go:64] FLAG: --manifest-url-header=""
Sep 17 17:41:01.128677 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:01.122737    2567 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 17 17:41:01.128677 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:01.122740    2567 flags.go:64] FLAG: --max-open-files="1000000"
Sep 17 17:41:01.128677 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:01.122745    2567 flags.go:64] FLAG: --max-pods="110"
Sep 17 17:41:01.128677 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:01.122749    2567 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 17 17:41:01.128677 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:01.122753    2567 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 17 17:41:01.128677 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:01.122756    2567 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 17 17:41:01.128677 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:01.122759    2567 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 17 17:41:01.129285 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:01.122762    2567 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 17 17:41:01.129285 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:01.122765    2567 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 17 17:41:01.129285 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:01.122768    2567 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 17 17:41:01.129285 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:01.122776    2567 flags.go:64] FLAG: --node-status-max-images="50"
Sep 17 17:41:01.129285 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:01.122779    2567 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 17 17:41:01.129285 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:01.122782    2567 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 17 17:41:01.129285 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:01.122785    2567 flags.go:64] FLAG: --pod-cidr=""
Sep 17 17:41:01.129285 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:01.122788    2567 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a6abe77af61c5f2e3985a83e7d43538aefc38b1b0444127da14ee0e8cf8f76fc"
Sep 17 17:41:01.129285 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:01.122794    2567 flags.go:64] FLAG: --pod-manifest-path=""
Sep 17 17:41:01.129285 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:01.122797    2567 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 17 17:41:01.129285 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:01.122800    2567 flags.go:64] FLAG: --pods-per-core="0"
Sep 17 17:41:01.129285 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:01.122803    2567 flags.go:64] FLAG: --port="10250"
Sep 17 17:41:01.129285 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:01.122806    2567 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 17 17:41:01.129285 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:01.122809    2567 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0e3c54f4649739401"
Sep 17 17:41:01.129285 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:01.122814    2567 flags.go:64] FLAG: --qos-reserved=""
Sep 17 17:41:01.129285 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:01.122817    2567 flags.go:64] FLAG: --read-only-port="10255"
Sep 17 17:41:01.129285 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:01.122820    2567 flags.go:64] FLAG: --register-node="true"
Sep 17 17:41:01.129285 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:01.122823    2567 flags.go:64] FLAG: --register-with-taints=""
Sep 17 17:41:01.129285 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:01.122827    2567 flags.go:64] FLAG: --registry-burst="10"
Sep 17 17:41:01.129285 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:01.122830    2567 flags.go:64] FLAG: --registry-qps="5"
Sep 17 17:41:01.129285 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:01.122833    2567 flags.go:64] FLAG: --reserved-cpus=""
Sep 17 17:41:01.129285 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:01.122836    2567 flags.go:64] FLAG: --reserved-memory=""
Sep 17 17:41:01.129285 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:01.122842    2567 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 17 17:41:01.129285 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:01.122845    2567 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 17 17:41:01.129285 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:01.122848    2567 flags.go:64] FLAG: --rotate-certificates="false"
Sep 17 17:41:01.129949 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:01.122851    2567 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 17 17:41:01.129949 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:01.122854    2567 flags.go:64] FLAG: --runonce="false"
Sep 17 17:41:01.129949 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:01.122857    2567 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 17 17:41:01.129949 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:01.122860    2567 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 17 17:41:01.129949 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:01.122865    2567 flags.go:64] FLAG: --seccomp-default="false"
Sep 17 17:41:01.129949 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:01.122868    2567 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 17 17:41:01.129949 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:01.122871    2567 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 17 17:41:01.129949 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:01.122875    2567 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 17 17:41:01.129949 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:01.122878    2567 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 17 17:41:01.129949 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:01.122881    2567 flags.go:64] FLAG: --storage-driver-password="root"
Sep 17 17:41:01.129949 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:01.122884    2567 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 17 17:41:01.129949 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:01.122887    2567 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 17 17:41:01.129949 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:01.122890    2567 flags.go:64] FLAG: --storage-driver-user="root"
Sep 17 17:41:01.129949 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:01.122893    2567 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 17 17:41:01.129949 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:01.122896    2567 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 17 17:41:01.129949 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:01.122900    2567 flags.go:64] FLAG: --system-cgroups=""
Sep 17 17:41:01.129949 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:01.122904    2567 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 17 17:41:01.129949 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:01.122909    2567 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 17 17:41:01.129949 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:01.122912    2567 flags.go:64] FLAG: --tls-cert-file=""
Sep 17 17:41:01.129949 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:01.122915    2567 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 17 17:41:01.129949 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:01.122920    2567 flags.go:64] FLAG: --tls-min-version=""
Sep 17 17:41:01.129949 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:01.122924    2567 flags.go:64] FLAG: --tls-private-key-file=""
Sep 17 17:41:01.129949 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:01.122927    2567 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 17 17:41:01.129949 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:01.122930    2567 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 17 17:41:01.129949 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:01.122933    2567 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 17 17:41:01.130603 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:01.122936    2567 flags.go:64] FLAG: --v="2"
Sep 17 17:41:01.130603 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:01.122940    2567 flags.go:64] FLAG: --version="false"
Sep 17 17:41:01.130603 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:01.122944    2567 flags.go:64] FLAG: --vmodule=""
Sep 17 17:41:01.130603 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:01.122948    2567 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 17 17:41:01.130603 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:01.122951    2567 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 17 17:41:01.130603 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.123073    2567 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 17:41:01.130603 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.123077    2567 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 17:41:01.130603 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.123080    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 17:41:01.130603 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.123082    2567 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 17:41:01.130603 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.123085    2567 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 17:41:01.130603 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.123088    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 17:41:01.130603 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.123093    2567 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 17:41:01.130603 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.123096    2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 17:41:01.130603 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.123098    2567 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 17:41:01.130603 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.123101    2567 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 17:41:01.130603 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.123104    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 17:41:01.130603 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.123107    2567 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 17:41:01.130603 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.123110    2567 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 17:41:01.130603 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.123113    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 17:41:01.130603 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.123115    2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 17:41:01.130603 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.123118    2567 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 17:41:01.131113 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.123121    2567 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 17:41:01.131113 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.123124    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 17:41:01.131113 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.123127    2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 17:41:01.131113 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.123130    2567 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 17:41:01.131113 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.123132    2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 17:41:01.131113 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.123135    2567 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 17:41:01.131113 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.123138    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 17:41:01.131113 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.123142    2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 17:41:01.131113 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.123144    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 17:41:01.131113 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.123147    2567 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 17:41:01.131113 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.123150    2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 17:41:01.131113 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.123152    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 17:41:01.131113 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.123155    2567 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 17:41:01.131113 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.123158    2567 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 17:41:01.131113 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.123161    2567 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 17:41:01.131113 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.123164    2567 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 17:41:01.131113 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.123166    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 17:41:01.131113 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.123169    2567 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 17:41:01.131113 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.123171    2567 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 17:41:01.131113 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.123174    2567 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 17:41:01.131611 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.123176    2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 17:41:01.131611 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.123179    2567 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 17:41:01.131611 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.123184    2567 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 17:41:01.131611 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.123188    2567 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 17:41:01.131611 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.123191    2567 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 17:41:01.131611 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.123194    2567 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 17:41:01.131611 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.123196    2567 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 17:41:01.131611 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.123199    2567 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 17:41:01.131611 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.123201    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 17:41:01.131611 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.123204    2567 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 17:41:01.131611 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.123207    2567 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 17:41:01.131611 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.123209    2567 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 17:41:01.131611 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.123212    2567 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 17:41:01.131611 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.123215    2567 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 17:41:01.131611 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.123219    2567 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 17:41:01.131611 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.123223    2567 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 17:41:01.131611 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.123226    2567 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 17:41:01.131611 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.123228    2567 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 17:41:01.131611 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.123231    2567 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 17:41:01.132078 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.123235    2567 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 17:41:01.132078 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.123238    2567 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 17:41:01.132078 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.123241    2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 17:41:01.132078 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.123243    2567 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 17:41:01.132078 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.123246    2567 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 17:41:01.132078 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.123248    2567 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 17:41:01.132078 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.123251    2567 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 17:41:01.132078 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.123254    2567 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 17:41:01.132078 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.123256    2567 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 17:41:01.132078 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.123259    2567 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 17:41:01.132078 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.123261    2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 17:41:01.132078 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.123264    2567 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 17:41:01.132078 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.123266    2567 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 17:41:01.132078 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.123269    2567 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 17:41:01.132078 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.123272    2567 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 17:41:01.132078 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.123276    2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 17:41:01.132078 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.123278    2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 17:41:01.132078 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.123281    2567 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 17:41:01.132078 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.123284    2567 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 17:41:01.132078 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.123286    2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 17:41:01.132588 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.123289    2567 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 17:41:01.132588 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.123292    2567 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 17:41:01.132588 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.123294    2567 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 17:41:01.132588 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.123297    2567 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 17:41:01.132588 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.123300    2567 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 17:41:01.132588 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.123302    2567 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 17:41:01.132588 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.123305    2567 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 17:41:01.132588 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.123307    2567 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 17:41:01.132588 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.123310    2567 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 17:41:01.132588 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.123313    2567 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 17:41:01.132588 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.123316    2567 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 17:41:01.132588 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.123318    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 17:41:01.132588 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.123323    2567 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 17:41:01.132588 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.123326    2567 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 17:41:01.132588 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.123329    2567 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 17:41:01.132588 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.123332    2567 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 17:41:01.132588 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.123336    2567 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 17:41:01.132588 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.123338    2567 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 17:41:01.132588 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.123341    2567 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 17:41:01.133058 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.123344    2567 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 17:41:01.133058 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.123347    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 17:41:01.133058 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.123350    2567 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 17:41:01.133058 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.123353    2567 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 17:41:01.133058 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.123356    2567 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 17:41:01.133058 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:01.123918    2567 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 17 17:41:01.133058 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:01.130848    2567 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 17 17:41:01.133058 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:01.130864    2567 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 17 17:41:01.133058 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:01.130879    2567 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 17 17:41:01.133058 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:01.130885    2567 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 17 17:41:01.133058 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.130940    2567 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 17:41:01.133058 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.130946    2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 17:41:01.133058 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.130949    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 17:41:01.133058 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.130953    2567 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 17:41:01.133058 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.130956    2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 17:41:01.133436 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.130959    2567 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 17:41:01.133436 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.130962    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 17:41:01.133436 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.130965    2567 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 17:41:01.133436 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.130968    2567 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 17:41:01.133436 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.130971    2567 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 17:41:01.133436 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.130973    2567 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 17:41:01.133436 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.130976    2567 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 17:41:01.133436 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.130979    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 17:41:01.133436 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.130982    2567 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 17:41:01.133436 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.130984    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 17:41:01.133436 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.130987    2567 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 17:41:01.133436 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.130989    2567 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 17:41:01.133436 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.130992    2567 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 17:41:01.133436 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.130995    2567 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 17:41:01.133436 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.130997    2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 17:41:01.133436 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.131000    2567 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 17:41:01.133436 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.131003    2567 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 17:41:01.133436 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.131005    2567 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 17:41:01.133436 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.131008    2567 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 17:41:01.133436 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.131011    2567 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 17:41:01.133987 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.131013    2567 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 17:41:01.133987 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.131016    2567 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 17:41:01.133987 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.131019    2567 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 17:41:01.133987 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.131021    2567 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 17:41:01.133987 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.131026    2567 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 17:41:01.133987 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.131028    2567 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 17:41:01.133987 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.131031    2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 17:41:01.133987 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.131033    2567 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 17:41:01.133987 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.131036    2567 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 17:41:01.133987 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.131039    2567 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 17:41:01.133987 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.131042    2567 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 17:41:01.133987 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.131045    2567 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 17:41:01.133987 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.131047    2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 17:41:01.133987 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.131050    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 17:41:01.133987 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.131053    2567 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 17:41:01.133987 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.131056    2567 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 17:41:01.133987 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.131058    2567 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 17:41:01.133987 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.131062    2567 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 17:41:01.133987 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.131065    2567 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 17:41:01.133987 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.131068    2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 17:41:01.134720 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.131071    2567 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 17:41:01.134720 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.131073    2567 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 17:41:01.134720 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.131076    2567 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 17:41:01.134720 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.131079    2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 17:41:01.134720 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.131081    2567 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 17:41:01.134720 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.131084    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 17:41:01.134720 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.131087    2567 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 17:41:01.134720 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.131091    2567 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 17:41:01.134720 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.131094    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 17:41:01.134720 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.131097    2567 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 17:41:01.134720 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.131099    2567 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 17:41:01.134720 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.131102    2567 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 17:41:01.134720 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.131105    2567 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 17:41:01.134720 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.131108    2567 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 17:41:01.134720 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.131110    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 17:41:01.134720 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.131113    2567 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 17:41:01.134720 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.131116    2567 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 17:41:01.134720 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.131119    2567 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 17:41:01.134720 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.131122    2567 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 17:41:01.135372 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.131126    2567 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 17:41:01.135372 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.131128    2567 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 17:41:01.135372 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.131131    2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 17:41:01.135372 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.131134    2567 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 17:41:01.135372 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.131136    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 17:41:01.135372 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.131139    2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 17:41:01.135372 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.131142    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 17:41:01.135372 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.131144    2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 17:41:01.135372 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.131147    2567 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 17:41:01.135372 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.131149    2567 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 17:41:01.135372 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.131152    2567 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 17:41:01.135372 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.131154    2567 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 17:41:01.135372 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.131157    2567 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 17:41:01.135372 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.131160    2567 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 17:41:01.135372 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.131162    2567 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 17:41:01.135372 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.131165    2567 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 17:41:01.135372 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.131168    2567 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 17:41:01.135372 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.131170    2567 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 17:41:01.135372 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.131173    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 17:41:01.135372 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.131175    2567 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 17:41:01.135888 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.131178    2567 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 17:41:01.135888 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.131180    2567 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 17:41:01.135888 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.131183    2567 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 17:41:01.135888 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.131185    2567 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 17:41:01.135888 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.131188    2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 17:41:01.135888 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.131190    2567 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 17:41:01.135888 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.131193    2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 17:41:01.135888 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.131195    2567 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 17:41:01.135888 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.131198    2567 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 17:41:01.135888 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.131211    2567 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 17:41:01.135888 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.131214    2567 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 17:41:01.135888 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.131220    2567 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 17:41:01.135888 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.131225    2567 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 17:41:01.135888 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.131228    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 17:41:01.135888 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.131231    2567 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 17:41:01.136309 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:01.131236    2567 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 17 17:41:01.136309 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.131349    2567 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 17:41:01.136309 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.131355    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 17:41:01.136309 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.131358    2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 17:41:01.136309 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.131360    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 17:41:01.136309 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.131363    2567 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 17:41:01.136309 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.131366    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 17:41:01.136309 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.131369    2567 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 17:41:01.136309 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.131371    2567 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 17:41:01.136309 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.131374    2567 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 17:41:01.136309 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.131377    2567 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 17:41:01.136309 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.131379    2567 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 17:41:01.136309 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.131382    2567 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 17:41:01.136309 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.131385    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 17:41:01.136309 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.131387    2567 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 17:41:01.136710 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.131390    2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 17:41:01.136710 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.131392    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 17:41:01.136710 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.131395    2567 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 17:41:01.136710 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.131397    2567 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 17:41:01.136710 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.131400    2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 17:41:01.136710 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.131402    2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 17:41:01.136710 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.131405    2567 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 17:41:01.136710 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.131409    2567 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 17:41:01.136710 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.131413    2567 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 17:41:01.136710 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.131416    2567 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 17:41:01.136710 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.131419    2567 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 17:41:01.136710 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.131422    2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 17:41:01.136710 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.131426    2567 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 17:41:01.136710 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.131428    2567 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 17:41:01.136710 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.131432    2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 17:41:01.136710 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.131435    2567 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 17:41:01.136710 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.131438    2567 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 17:41:01.136710 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.131440    2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 17:41:01.136710 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.131443    2567 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 17:41:01.137175 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.131446    2567 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 17:41:01.137175 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.131448    2567 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 17:41:01.137175 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.131451    2567 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 17:41:01.137175 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.131453    2567 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 17:41:01.137175 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.131456    2567 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 17:41:01.137175 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.131458    2567 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 17:41:01.137175 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.131460    2567 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 17:41:01.137175 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.131463    2567 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 17:41:01.137175 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.131466    2567 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 17:41:01.137175 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.131468    2567 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 17:41:01.137175 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.131471    2567 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 17:41:01.137175 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.131473    2567 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 17:41:01.137175 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.131476    2567 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 17:41:01.137175 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.131478    2567 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 17:41:01.137175 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.131481    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 17:41:01.137175 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.131484    2567 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 17:41:01.137175 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.131486    2567 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 17:41:01.137175 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.131489    2567 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 17:41:01.137175 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.131492    2567 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 17:41:01.137175 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.131494    2567 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 17:41:01.137674 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.131497    2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 17:41:01.137674 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.131499    2567 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 17:41:01.137674 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.131502    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 17:41:01.137674 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.131504    2567 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 17:41:01.137674 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.131507    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 17:41:01.137674 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.131510    2567 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 17:41:01.137674 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.131512    2567 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 17:41:01.137674 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.131515    2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 17:41:01.137674 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.131519    2567 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 17:41:01.137674 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.131522    2567 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 17:41:01.137674 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.131525    2567 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 17:41:01.137674 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.131542    2567 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 17:41:01.137674 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.131545    2567 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 17:41:01.137674 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.131547    2567 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 17:41:01.137674 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.131550    2567 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 17:41:01.137674 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.131553    2567 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 17:41:01.137674 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.131555    2567 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 17:41:01.137674 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.131558    2567 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 17:41:01.137674 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.131561    2567 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 17:41:01.138135 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.131564    2567 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 17:41:01.138135 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.131567    2567 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 17:41:01.138135 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.131570    2567 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 17:41:01.138135 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.131572    2567 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 17:41:01.138135 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.131576    2567 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 17:41:01.138135 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.131580    2567 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 17:41:01.138135 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.131583    2567 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 17:41:01.138135 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.131586    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 17:41:01.138135 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.131589    2567 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 17:41:01.138135 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.131592    2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 17:41:01.138135 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.131594    2567 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 17:41:01.138135 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.131597    2567 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 17:41:01.138135 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.131600    2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 17:41:01.138135 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.131603    2567 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 17:41:01.138135 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.131605    2567 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 17:41:01.138135 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.131607    2567 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 17:41:01.138135 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.131610    2567 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 17:41:01.138135 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.131613    2567 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 17:41:01.138135 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.131616    2567 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 17:41:01.138135 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.131618    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 17:41:01.138751 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.131621    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 17:41:01.138751 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.131624    2567 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 17:41:01.138751 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.131627    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 17:41:01.138751 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.131630    2567 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 17:41:01.138751 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.131633    2567 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 17:41:01.138751 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.131636    2567 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 17:41:01.138751 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:01.131638    2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 17:41:01.138751 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:01.131643    2567 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 17 17:41:01.138751 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:01.131749    2567 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 17 17:41:01.138751 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:01.133996    2567 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 17 17:41:01.138751 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:01.136313    2567 server.go:1019] "Starting client certificate rotation"
Sep 17 17:41:01.138751 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:01.136414    2567 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 17 17:41:01.138751 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:01.136455    2567 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 17 17:41:01.139292 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:01.139276    2567 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 17 17:41:01.140454 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:01.140436    2567 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 17 17:41:01.144849 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:01.144830    2567 log.go:25] "Validated CRI v1 runtime API"
Sep 17 17:41:01.146379 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:01.146364    2567 log.go:25] "Validated CRI v1 image API"
Sep 17 17:41:01.147144 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:01.147130    2567 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 17 17:41:01.149294 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:01.149274    2567 fs.go:135] Filesystem UUIDs: map[234740ee-2efa-482e-91b4-b069c79a7c3e:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 b4be79c4-64a9-4e10-978d-345304221b68:/dev/nvme0n1p3]
Sep 17 17:41:01.149365 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:01.149294    2567 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 17 17:41:01.154575 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:01.154438    2567 manager.go:216] Machine: {Timestamp:2026-09-17 17:41:01.153967889 +0000 UTC m=+2.177573037 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3101327 MemoryCapacity:33164480512 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec25a73c1faeacc9f37c49a2e3ffeed5 SystemUUID:ec25a73c-1fae-acc9-f37c-49a2e3ffeed5 BootID:ada3a6c5-1958-4767-997d-7e6651745c9d Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582242304 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:e7:3b:b9:77 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:e7:3b:b9:77 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:0e:b0:45:5b:dc:1e Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164480512 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 17 17:41:01.154575 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:01.154567    2567 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 17 17:41:01.154741 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:01.154668    2567 manager.go:232] Version: {KernelVersion:5.14.0-570.141.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260914-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 17 17:41:01.156311 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:01.156283    2567 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 17 17:41:01.156496 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:01.156312    2567 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-139-237.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 17 17:41:01.158838 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:01.158824    2567 topology_manager.go:138] "Creating topology manager with none policy"
Sep 17 17:41:01.158906 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:01.158843    2567 container_manager_linux.go:309] "Creating device plugin manager"
Sep 17 17:41:01.158906 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:01.158856    2567 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 17 17:41:01.158906 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:01.158871    2567 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 17 17:41:01.159033 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:01.158958    2567 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 17 17:41:01.163211 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:01.163197    2567 state_mem.go:36] "Initialized new in-memory state store"
Sep 17 17:41:01.163345 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:01.163333    2567 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 17 17:41:01.163400 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:01.163384    2567 kubelet.go:486] "Attempting to sync node with API server"
Sep 17 17:41:01.163446 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:01.163404    2567 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 17 17:41:01.163446 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:01.163424    2567 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 17 17:41:01.163446 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:01.163438    2567 kubelet.go:398] "Adding apiserver pod source"
Sep 17 17:41:01.163580 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:01.163451    2567 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 17 17:41:01.163890 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:01.163875    2567 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 17 17:41:01.163949 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:01.163903    2567 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 17 17:41:01.166682 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:01.166661    2567 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.13-6.rhaos4.21.git6253162.el9" apiVersion="v1"
Sep 17 17:41:01.170357 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:01.170325    2567 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 17 17:41:01.170437 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:01.170379    2567 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 17 17:41:01.170437 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:01.170407    2567 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 17 17:41:01.171301 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:01.171288    2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 17 17:41:01.171352 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:01.171308    2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 17 17:41:01.171352 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:01.171316    2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 17 17:41:01.171352 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:01.171322    2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 17 17:41:01.171352 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:01.171331    2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 17 17:41:01.171352 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:01.171336    2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 17 17:41:01.171644 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:01.171635    2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 17 17:41:01.171682 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:01.171646    2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 17 17:41:01.171682 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:01.171653    2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 17 17:41:01.171682 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:01.171659    2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 17 17:41:01.171682 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:01.171667    2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 17 17:41:01.171682 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:01.171675    2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 17 17:41:01.171819 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:01.171709    2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 17 17:41:01.171819 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:01.171717    2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 17 17:41:01.174601 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:01.174585    2567 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-139-237.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 17 17:41:01.174720 ip-10-0-139-237 kubenswrapper[2567]: E0917 17:41:01.174701    2567 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-139-237.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 17 17:41:01.174762 ip-10-0-139-237 kubenswrapper[2567]: E0917 17:41:01.174699    2567 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 17 17:41:01.175966 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:01.175953    2567 server.go:1268] "Started kubelet"
Sep 17 17:41:01.176069 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:01.176022    2567 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 17 17:41:01.176117 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:01.176090    2567 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 17 17:41:01.176164 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:01.176129    2567 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 17 17:41:01.176936 ip-10-0-139-237 systemd[1]: Started Kubernetes Kubelet.
Sep 17 17:41:01.178006 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:01.177942    2567 server.go:310] "Adding debug handlers to kubelet server"
Sep 17 17:41:01.180208 ip-10-0-139-237 kubenswrapper[2567]: E0917 17:41:01.180176    2567 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 17 17:41:01.180292 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:01.180261    2567 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 17 17:41:01.184360 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:01.184346    2567 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 17 17:41:01.184429 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:01.184367    2567 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 17 17:41:01.184476 ip-10-0-139-237 kubenswrapper[2567]: E0917 17:41:01.184416    2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-237.ec2.internal\" not found"
Sep 17 17:41:01.184911 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:01.184894    2567 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 17 17:41:01.184911 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:01.184907    2567 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 17 17:41:01.185034 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:01.184927    2567 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 17 17:41:01.185096 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:01.185077    2567 reconstruct.go:97] "Volume reconstruction finished"
Sep 17 17:41:01.185096 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:01.185087    2567 reconciler.go:29] "Reconciler: start to sync state"
Sep 17 17:41:01.185333 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:01.185318    2567 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 17 17:41:01.185399 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:01.185335    2567 factory.go:55] Registering systemd factory
Sep 17 17:41:01.185399 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:01.185346    2567 factory.go:223] Registration of the systemd container factory successfully
Sep 17 17:41:01.185564 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:01.185551    2567 factory.go:153] Registering CRI-O factory
Sep 17 17:41:01.185612 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:01.185566    2567 factory.go:223] Registration of the crio container factory successfully
Sep 17 17:41:01.185612 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:01.185583    2567 factory.go:103] Registering Raw factory
Sep 17 17:41:01.185612 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:01.185596    2567 manager.go:1196] Started watching for new ooms in manager
Sep 17 17:41:01.186028 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:01.186014    2567 manager.go:349] Starting recovery of all containers
Sep 17 17:41:01.187500 ip-10-0-139-237 kubenswrapper[2567]: E0917 17:41:01.187478    2567 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 17 17:41:01.187614 ip-10-0-139-237 kubenswrapper[2567]: E0917 17:41:01.187556    2567 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-139-237.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 17 17:41:01.194756 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:01.194739    2567 manager.go:354] Recovery completed
Sep 17 17:41:01.206943 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:01.206914    2567 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-hxtgw"
Sep 17 17:41:01.208599 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:01.208582    2567 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-hxtgw"
Sep 17 17:41:01.214459 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:01.214444    2567 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 17:41:01.216218 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:01.216201    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-237.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 17:41:01.216303 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:01.216228    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-237.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 17:41:01.216303 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:01.216238    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-237.ec2.internal" event="NodeHasSufficientPID"
Sep 17 17:41:01.216795 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:01.216780    2567 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 17 17:41:01.216795 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:01.216793    2567 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 17 17:41:01.216908 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:01.216808    2567 state_mem.go:36] "Initialized new in-memory state store"
Sep 17 17:41:01.216908 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:01.216833    2567 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 17:41:01.217543 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:01.217514    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-237.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 17:41:01.217603 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:01.217556    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-237.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 17:41:01.217603 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:01.217571    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-237.ec2.internal" event="NodeHasSufficientPID"
Sep 17 17:41:01.218876 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:01.218863    2567 policy_none.go:49] "None policy: Start"
Sep 17 17:41:01.218935 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:01.218880    2567 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 17 17:41:01.218935 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:01.218890    2567 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 17 17:41:01.221107 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:01.221093    2567 policy_none.go:47] "Start"
Sep 17 17:41:01.261021 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:01.255101    2567 manager.go:337] "Starting Device Plugin manager"
Sep 17 17:41:01.261021 ip-10-0-139-237 kubenswrapper[2567]: E0917 17:41:01.255133    2567 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 17 17:41:01.261021 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:01.255143    2567 server.go:87] "Starting device plugin registration server"
Sep 17 17:41:01.261021 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:01.255375    2567 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 17 17:41:01.261021 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:01.255398    2567 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 17 17:41:01.261021 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:01.255488    2567 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 17 17:41:01.261021 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:01.255587    2567 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 17 17:41:01.261021 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:01.255596    2567 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 17 17:41:01.261021 ip-10-0-139-237 kubenswrapper[2567]: E0917 17:41:01.256047    2567 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 17 17:41:01.261021 ip-10-0-139-237 kubenswrapper[2567]: E0917 17:41:01.256094    2567 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-139-237.ec2.internal\" not found"
Sep 17 17:41:01.279233 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:01.279205    2567 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 17 17:41:01.280385 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:01.280369    2567 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 17 17:41:01.280448 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:01.280424    2567 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 17 17:41:01.280493 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:01.280450    2567 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 17 17:41:01.280557 ip-10-0-139-237 kubenswrapper[2567]: E0917 17:41:01.280490    2567 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 17 17:41:01.282946 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:01.282923    2567 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 17:41:01.355790 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:01.355764    2567 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 17:41:01.362858 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:01.362836    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-237.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 17:41:01.362984 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:01.362873    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-237.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 17:41:01.362984 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:01.362894    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-237.ec2.internal" event="NodeHasSufficientPID"
Sep 17 17:41:01.362984 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:01.362920    2567 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-139-237.ec2.internal"
Sep 17 17:41:01.373632 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:01.373585    2567 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-139-237.ec2.internal"
Sep 17 17:41:01.373632 ip-10-0-139-237 kubenswrapper[2567]: E0917 17:41:01.373605    2567 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-139-237.ec2.internal\": node \"ip-10-0-139-237.ec2.internal\" not found"
Sep 17 17:41:01.380751 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:01.380726    2567 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-237.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-139-237.ec2.internal"]
Sep 17 17:41:01.380803 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:01.380790    2567 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 17:41:01.381742 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:01.381729    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-237.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 17:41:01.381797 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:01.381751    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-237.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 17:41:01.381797 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:01.381761    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-237.ec2.internal" event="NodeHasSufficientPID"
Sep 17 17:41:01.383953 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:01.383940    2567 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 17:41:01.384101 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:01.384086    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-237.ec2.internal"
Sep 17 17:41:01.384157 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:01.384122    2567 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 17:41:01.384794 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:01.384780    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-237.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 17:41:01.384794 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:01.384787    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-237.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 17:41:01.384909 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:01.384810    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-237.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 17:41:01.384909 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:01.384810    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-237.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 17:41:01.384909 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:01.384846    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-237.ec2.internal" event="NodeHasSufficientPID"
Sep 17 17:41:01.384909 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:01.384825    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-237.ec2.internal" event="NodeHasSufficientPID"
Sep 17 17:41:01.385464 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:01.385447    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e678eb327b2639be46e3883082cf83ef-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-237.ec2.internal\" (UID: \"e678eb327b2639be46e3883082cf83ef\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-237.ec2.internal"
Sep 17 17:41:01.385520 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:01.385480    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e678eb327b2639be46e3883082cf83ef-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-237.ec2.internal\" (UID: \"e678eb327b2639be46e3883082cf83ef\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-237.ec2.internal"
Sep 17 17:41:01.385520 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:01.385505    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/840f4093b21dc0fac911784fe7877e0e-config\") pod \"kube-apiserver-proxy-ip-10-0-139-237.ec2.internal\" (UID: \"840f4093b21dc0fac911784fe7877e0e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-237.ec2.internal"
Sep 17 17:41:01.387033 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:01.387017    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-237.ec2.internal"
Sep 17 17:41:01.387111 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:01.387043    2567 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 17:41:01.387747 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:01.387730    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-237.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 17:41:01.387822 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:01.387760    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-237.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 17:41:01.387822 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:01.387772    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-237.ec2.internal" event="NodeHasSufficientPID"
Sep 17 17:41:01.401034 ip-10-0-139-237 kubenswrapper[2567]: E0917 17:41:01.401014    2567 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-237.ec2.internal\" not found" node="ip-10-0-139-237.ec2.internal"
Sep 17 17:41:01.405401 ip-10-0-139-237 kubenswrapper[2567]: E0917 17:41:01.405386    2567 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-237.ec2.internal\" not found" node="ip-10-0-139-237.ec2.internal"
Sep 17 17:41:01.409913 ip-10-0-139-237 kubenswrapper[2567]: E0917 17:41:01.409899    2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-237.ec2.internal\" not found"
Sep 17 17:41:01.486269 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:01.486241    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e678eb327b2639be46e3883082cf83ef-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-237.ec2.internal\" (UID: \"e678eb327b2639be46e3883082cf83ef\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-237.ec2.internal"
Sep 17 17:41:01.486367 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:01.486192    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e678eb327b2639be46e3883082cf83ef-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-237.ec2.internal\" (UID: \"e678eb327b2639be46e3883082cf83ef\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-237.ec2.internal"
Sep 17 17:41:01.486367 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:01.486302    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e678eb327b2639be46e3883082cf83ef-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-237.ec2.internal\" (UID: \"e678eb327b2639be46e3883082cf83ef\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-237.ec2.internal"
Sep 17 17:41:01.486367 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:01.486319    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/840f4093b21dc0fac911784fe7877e0e-config\") pod \"kube-apiserver-proxy-ip-10-0-139-237.ec2.internal\" (UID: \"840f4093b21dc0fac911784fe7877e0e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-237.ec2.internal"
Sep 17 17:41:01.486367 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:01.486346    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/840f4093b21dc0fac911784fe7877e0e-config\") pod \"kube-apiserver-proxy-ip-10-0-139-237.ec2.internal\" (UID: \"840f4093b21dc0fac911784fe7877e0e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-237.ec2.internal"
Sep 17 17:41:01.486487 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:01.486364    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e678eb327b2639be46e3883082cf83ef-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-237.ec2.internal\" (UID: \"e678eb327b2639be46e3883082cf83ef\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-237.ec2.internal"
Sep 17 17:41:01.510642 ip-10-0-139-237 kubenswrapper[2567]: E0917 17:41:01.510620    2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-237.ec2.internal\" not found"
Sep 17 17:41:01.611674 ip-10-0-139-237 kubenswrapper[2567]: E0917 17:41:01.611646    2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-237.ec2.internal\" not found"
Sep 17 17:41:01.704318 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:01.704257    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-237.ec2.internal"
Sep 17 17:41:01.708210 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:01.708188    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-237.ec2.internal"
Sep 17 17:41:01.713993 ip-10-0-139-237 kubenswrapper[2567]: E0917 17:41:01.712669    2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-237.ec2.internal\" not found"
Sep 17 17:41:01.813883 ip-10-0-139-237 kubenswrapper[2567]: E0917 17:41:01.813707    2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-237.ec2.internal\" not found"
Sep 17 17:41:01.914239 ip-10-0-139-237 kubenswrapper[2567]: E0917 17:41:01.914214    2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-237.ec2.internal\" not found"
Sep 17 17:41:02.014758 ip-10-0-139-237 kubenswrapper[2567]: E0917 17:41:02.014686    2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-237.ec2.internal\" not found"
Sep 17 17:41:02.115237 ip-10-0-139-237 kubenswrapper[2567]: E0917 17:41:02.115210    2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-237.ec2.internal\" not found"
Sep 17 17:41:02.136385 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:02.136357    2567 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 17 17:41:02.136777 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:02.136544    2567 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 17 17:41:02.213988 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:02.213948    2567 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-16 17:36:01 +0000 UTC" deadline="2028-03-19 05:53:55.237534961 +0000 UTC"
Sep 17 17:41:02.213988 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:02.213978    2567 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13164h12m53.023560981s"
Sep 17 17:41:02.215947 ip-10-0-139-237 kubenswrapper[2567]: E0917 17:41:02.215928    2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-237.ec2.internal\" not found"
Sep 17 17:41:02.226989 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:02.226971    2567 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 17:41:02.285088 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:02.285025    2567 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-237.ec2.internal"
Sep 17 17:41:02.307937 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:02.307919    2567 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 17 17:41:02.310450 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:02.310437    2567 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-237.ec2.internal"
Sep 17 17:41:02.329266 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:02.329248    2567 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 17 17:41:02.365009 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:02.364990    2567 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 17:41:02.410518 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:02.410494    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode678eb327b2639be46e3883082cf83ef.slice/crio-73d0a869b4d16f36a006af6e0cd7afecfa6f56933db3a9323727750d324aa71c WatchSource:0}: Error finding container 73d0a869b4d16f36a006af6e0cd7afecfa6f56933db3a9323727750d324aa71c: Status 404 returned error can't find the container with id 73d0a869b4d16f36a006af6e0cd7afecfa6f56933db3a9323727750d324aa71c
Sep 17 17:41:02.411403 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:02.411388    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod840f4093b21dc0fac911784fe7877e0e.slice/crio-4e2546499cbd4b2787c66a780c1f389feb4c38cdee7b5d6ff62f00888df7e95f WatchSource:0}: Error finding container 4e2546499cbd4b2787c66a780c1f389feb4c38cdee7b5d6ff62f00888df7e95f: Status 404 returned error can't find the container with id 4e2546499cbd4b2787c66a780c1f389feb4c38cdee7b5d6ff62f00888df7e95f
Sep 17 17:41:02.417274 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:02.417257    2567 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 17:41:02.566016 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:02.565937    2567 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 17:41:03.164450 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:03.164411    2567 apiserver.go:52] "Watching apiserver"
Sep 17 17:41:03.172509 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:03.172472    2567 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 17 17:41:03.172992 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:03.172965    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-237.ec2.internal","openshift-multus/multus-dsxjr","openshift-multus/network-metrics-daemon-xg2wr","openshift-ovn-kubernetes/ovnkube-node-vwtvz","kube-system/global-pull-secret-syncer-vjstm","openshift-cluster-node-tuning-operator/tuned-vvjl9","openshift-image-registry/node-ca-cfn7g","openshift-multus/multus-additional-cni-plugins-ptjgb","openshift-network-diagnostics/network-check-target-4cb4f","openshift-network-operator/iptables-alerter-zrbgv","kube-system/konnectivity-agent-j5jw8","kube-system/kube-apiserver-proxy-ip-10-0-139-237.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6xdld","openshift-dns/node-resolver-454w2"]
Sep 17 17:41:03.175488 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:03.175465    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-cfn7g"
Sep 17 17:41:03.177837 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:03.177816    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6xdld"
Sep 17 17:41:03.178429 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:03.178244    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 17 17:41:03.178429 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:03.178322    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-tqwnt\""
Sep 17 17:41:03.178429 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:03.178331    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 17 17:41:03.178713 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:03.178611    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 17 17:41:03.180174 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:03.180151    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 17 17:41:03.180270 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:03.180240    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 17 17:41:03.180329 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:03.180161    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 17 17:41:03.180599 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:03.180582    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-l9j92\""
Sep 17 17:41:03.180677 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:03.180643    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xg2wr"
Sep 17 17:41:03.180738 ip-10-0-139-237 kubenswrapper[2567]: E0917 17:41:03.180705    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xg2wr" podUID="5ab7aae3-053b-4e36-9c4a-c523f0cfb615"
Sep 17 17:41:03.182627 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:03.182607    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-vwtvz"
Sep 17 17:41:03.184912 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:03.184891    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vjstm"
Sep 17 17:41:03.185002 ip-10-0-139-237 kubenswrapper[2567]: E0917 17:41:03.184958    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vjstm" podUID="db73a8fe-d34c-41c8-8ba4-8040b4bacf13"
Sep 17 17:41:03.185067 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:03.185022    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 17 17:41:03.185502 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:03.185485    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 17 17:41:03.185673 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:03.185650    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 17 17:41:03.185920 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:03.185802    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-t8zn5\""
Sep 17 17:41:03.185920 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:03.185851    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 17 17:41:03.186034 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:03.184902    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 17 17:41:03.186086 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:03.186076    2567 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 17 17:41:03.186399 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:03.186368    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 17 17:41:03.187676 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:03.187343    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-vvjl9"
Sep 17 17:41:03.189962 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:03.189942    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 17 17:41:03.190067 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:03.190033    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 17 17:41:03.190128 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:03.190063    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-c5k52\""
Sep 17 17:41:03.190241 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:03.190226    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4cb4f"
Sep 17 17:41:03.190313 ip-10-0-139-237 kubenswrapper[2567]: E0917 17:41:03.190283    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4cb4f" podUID="6d0a7973-cc86-4fda-9056-11b5923bd14c"
Sep 17 17:41:03.192662 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:03.192323    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-454w2"
Sep 17 17:41:03.195062 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:03.194776    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-dsxjr"
Sep 17 17:41:03.195201 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:03.195168    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/db73a8fe-d34c-41c8-8ba4-8040b4bacf13-original-pull-secret\") pod \"global-pull-secret-syncer-vjstm\" (UID: \"db73a8fe-d34c-41c8-8ba4-8040b4bacf13\") " pod="kube-system/global-pull-secret-syncer-vjstm"
Sep 17 17:41:03.195271 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:03.195213    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/451c115d-30f1-4679-be41-ff6d2beedaf8-etc-sysctl-conf\") pod \"tuned-vvjl9\" (UID: \"451c115d-30f1-4679-be41-ff6d2beedaf8\") " pod="openshift-cluster-node-tuning-operator/tuned-vvjl9"
Sep 17 17:41:03.195271 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:03.195236    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/7e56ce66-8c6f-4fa0-9560-6210a25593fa-sys-fs\") pod \"aws-ebs-csi-driver-node-6xdld\" (UID: \"7e56ce66-8c6f-4fa0-9560-6210a25593fa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6xdld"
Sep 17 17:41:03.195271 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:03.195259    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5ab7aae3-053b-4e36-9c4a-c523f0cfb615-metrics-certs\") pod \"network-metrics-daemon-xg2wr\" (UID: \"5ab7aae3-053b-4e36-9c4a-c523f0cfb615\") " pod="openshift-multus/network-metrics-daemon-xg2wr"
Sep 17 17:41:03.195421 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:03.195282    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0a5f3756-68b8-4e1d-97fc-75a9e6e6fb27-run-openvswitch\") pod \"ovnkube-node-vwtvz\" (UID: \"0a5f3756-68b8-4e1d-97fc-75a9e6e6fb27\") " pod="openshift-ovn-kubernetes/ovnkube-node-vwtvz"
Sep 17 17:41:03.195421 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:03.195304    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/0a5f3756-68b8-4e1d-97fc-75a9e6e6fb27-ovn-node-metrics-cert\") pod \"ovnkube-node-vwtvz\" (UID: \"0a5f3756-68b8-4e1d-97fc-75a9e6e6fb27\") " pod="openshift-ovn-kubernetes/ovnkube-node-vwtvz"
Sep 17 17:41:03.195421 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:03.195329    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/db73a8fe-d34c-41c8-8ba4-8040b4bacf13-kubelet-config\") pod \"global-pull-secret-syncer-vjstm\" (UID: \"db73a8fe-d34c-41c8-8ba4-8040b4bacf13\") " pod="kube-system/global-pull-secret-syncer-vjstm"
Sep 17 17:41:03.195421 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:03.195339    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 17 17:41:03.195421 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:03.195351    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/451c115d-30f1-4679-be41-ff6d2beedaf8-etc-kubernetes\") pod \"tuned-vvjl9\" (UID: \"451c115d-30f1-4679-be41-ff6d2beedaf8\") " pod="openshift-cluster-node-tuning-operator/tuned-vvjl9"
Sep 17 17:41:03.195421 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:03.195367    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 17 17:41:03.195421 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:03.195372    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/451c115d-30f1-4679-be41-ff6d2beedaf8-etc-sysctl-d\") pod \"tuned-vvjl9\" (UID: \"451c115d-30f1-4679-be41-ff6d2beedaf8\") " pod="openshift-cluster-node-tuning-operator/tuned-vvjl9"
Sep 17 17:41:03.195421 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:03.195394    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/451c115d-30f1-4679-be41-ff6d2beedaf8-etc-systemd\") pod \"tuned-vvjl9\" (UID: \"451c115d-30f1-4679-be41-ff6d2beedaf8\") " pod="openshift-cluster-node-tuning-operator/tuned-vvjl9"
Sep 17 17:41:03.195421 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:03.195419    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xbxgz\" (UniqueName: \"kubernetes.io/projected/e45ad616-0131-4f4e-adf4-b567b76edc38-kube-api-access-xbxgz\") pod \"node-ca-cfn7g\" (UID: \"e45ad616-0131-4f4e-adf4-b567b76edc38\") " pod="openshift-image-registry/node-ca-cfn7g"
Sep 17 17:41:03.195843 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:03.195443    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/7e56ce66-8c6f-4fa0-9560-6210a25593fa-socket-dir\") pod \"aws-ebs-csi-driver-node-6xdld\" (UID: \"7e56ce66-8c6f-4fa0-9560-6210a25593fa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6xdld"
Sep 17 17:41:03.195843 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:03.195465    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/7e56ce66-8c6f-4fa0-9560-6210a25593fa-etc-selinux\") pod \"aws-ebs-csi-driver-node-6xdld\" (UID: \"7e56ce66-8c6f-4fa0-9560-6210a25593fa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6xdld"
Sep 17 17:41:03.195843 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:03.195490    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gb66b\" (UniqueName: \"kubernetes.io/projected/7e56ce66-8c6f-4fa0-9560-6210a25593fa-kube-api-access-gb66b\") pod \"aws-ebs-csi-driver-node-6xdld\" (UID: \"7e56ce66-8c6f-4fa0-9560-6210a25593fa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6xdld"
Sep 17 17:41:03.195843 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:03.195514    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0a5f3756-68b8-4e1d-97fc-75a9e6e6fb27-host-slash\") pod \"ovnkube-node-vwtvz\" (UID: \"0a5f3756-68b8-4e1d-97fc-75a9e6e6fb27\") " pod="openshift-ovn-kubernetes/ovnkube-node-vwtvz"
Sep 17 17:41:03.195843 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:03.195571    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0a5f3756-68b8-4e1d-97fc-75a9e6e6fb27-run-systemd\") pod \"ovnkube-node-vwtvz\" (UID: \"0a5f3756-68b8-4e1d-97fc-75a9e6e6fb27\") " pod="openshift-ovn-kubernetes/ovnkube-node-vwtvz"
Sep 17 17:41:03.195843 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:03.195629    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0a5f3756-68b8-4e1d-97fc-75a9e6e6fb27-host-cni-netd\") pod \"ovnkube-node-vwtvz\" (UID: \"0a5f3756-68b8-4e1d-97fc-75a9e6e6fb27\") " pod="openshift-ovn-kubernetes/ovnkube-node-vwtvz"
Sep 17 17:41:03.195843 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:03.195656    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0a5f3756-68b8-4e1d-97fc-75a9e6e6fb27-run-ovn\") pod \"ovnkube-node-vwtvz\" (UID: \"0a5f3756-68b8-4e1d-97fc-75a9e6e6fb27\") " pod="openshift-ovn-kubernetes/ovnkube-node-vwtvz"
Sep 17 17:41:03.195843 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:03.195682    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0a5f3756-68b8-4e1d-97fc-75a9e6e6fb27-log-socket\") pod \"ovnkube-node-vwtvz\" (UID: \"0a5f3756-68b8-4e1d-97fc-75a9e6e6fb27\") " pod="openshift-ovn-kubernetes/ovnkube-node-vwtvz"
Sep 17 17:41:03.195843 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:03.195710    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/7e56ce66-8c6f-4fa0-9560-6210a25593fa-registration-dir\") pod \"aws-ebs-csi-driver-node-6xdld\" (UID: \"7e56ce66-8c6f-4fa0-9560-6210a25593fa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6xdld"
Sep 17 17:41:03.195843 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:03.195724    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-xqkf7\""
Sep 17 17:41:03.195843 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:03.195734    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0a5f3756-68b8-4e1d-97fc-75a9e6e6fb27-systemd-units\") pod \"ovnkube-node-vwtvz\" (UID: \"0a5f3756-68b8-4e1d-97fc-75a9e6e6fb27\") " pod="openshift-ovn-kubernetes/ovnkube-node-vwtvz"
Sep 17 17:41:03.195843 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:03.195758    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kg27h\" (UniqueName: \"kubernetes.io/projected/0a5f3756-68b8-4e1d-97fc-75a9e6e6fb27-kube-api-access-kg27h\") pod \"ovnkube-node-vwtvz\" (UID: \"0a5f3756-68b8-4e1d-97fc-75a9e6e6fb27\") " pod="openshift-ovn-kubernetes/ovnkube-node-vwtvz"
Sep 17 17:41:03.195843 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:03.195782    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/451c115d-30f1-4679-be41-ff6d2beedaf8-etc-sysconfig\") pod \"tuned-vvjl9\" (UID: \"451c115d-30f1-4679-be41-ff6d2beedaf8\") " pod="openshift-cluster-node-tuning-operator/tuned-vvjl9"
Sep 17 17:41:03.195843 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:03.195806    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/451c115d-30f1-4679-be41-ff6d2beedaf8-var-lib-kubelet\") pod \"tuned-vvjl9\" (UID: \"451c115d-30f1-4679-be41-ff6d2beedaf8\") " pod="openshift-cluster-node-tuning-operator/tuned-vvjl9"
Sep 17 17:41:03.195843 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:03.195828    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e45ad616-0131-4f4e-adf4-b567b76edc38-host\") pod \"node-ca-cfn7g\" (UID: \"e45ad616-0131-4f4e-adf4-b567b76edc38\") " pod="openshift-image-registry/node-ca-cfn7g"
Sep 17 17:41:03.196548 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:03.195852    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0a5f3756-68b8-4e1d-97fc-75a9e6e6fb27-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-vwtvz\" (UID: \"0a5f3756-68b8-4e1d-97fc-75a9e6e6fb27\") " pod="openshift-ovn-kubernetes/ovnkube-node-vwtvz"
Sep 17 17:41:03.196548 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:03.195875    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/0a5f3756-68b8-4e1d-97fc-75a9e6e6fb27-ovnkube-script-lib\") pod \"ovnkube-node-vwtvz\" (UID: \"0a5f3756-68b8-4e1d-97fc-75a9e6e6fb27\") " pod="openshift-ovn-kubernetes/ovnkube-node-vwtvz"
Sep 17 17:41:03.196548 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:03.195895    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/451c115d-30f1-4679-be41-ff6d2beedaf8-sys\") pod \"tuned-vvjl9\" (UID: \"451c115d-30f1-4679-be41-ff6d2beedaf8\") " pod="openshift-cluster-node-tuning-operator/tuned-vvjl9"
Sep 17 17:41:03.196548 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:03.195960    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/451c115d-30f1-4679-be41-ff6d2beedaf8-etc-tuned\") pod \"tuned-vvjl9\" (UID: \"451c115d-30f1-4679-be41-ff6d2beedaf8\") " pod="openshift-cluster-node-tuning-operator/tuned-vvjl9"
Sep 17 17:41:03.196548 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:03.196036    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/451c115d-30f1-4679-be41-ff6d2beedaf8-tmp\") pod \"tuned-vvjl9\" (UID: \"451c115d-30f1-4679-be41-ff6d2beedaf8\") " pod="openshift-cluster-node-tuning-operator/tuned-vvjl9"
Sep 17 17:41:03.196548 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:03.196072    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mmpqq\" (UniqueName: \"kubernetes.io/projected/451c115d-30f1-4679-be41-ff6d2beedaf8-kube-api-access-mmpqq\") pod \"tuned-vvjl9\" (UID: \"451c115d-30f1-4679-be41-ff6d2beedaf8\") " pod="openshift-cluster-node-tuning-operator/tuned-vvjl9"
Sep 17 17:41:03.196548 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:03.196098    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0a5f3756-68b8-4e1d-97fc-75a9e6e6fb27-host-run-ovn-kubernetes\") pod \"ovnkube-node-vwtvz\" (UID: \"0a5f3756-68b8-4e1d-97fc-75a9e6e6fb27\") " pod="openshift-ovn-kubernetes/ovnkube-node-vwtvz"
Sep 17 17:41:03.196548 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:03.196151    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/0a5f3756-68b8-4e1d-97fc-75a9e6e6fb27-env-overrides\") pod \"ovnkube-node-vwtvz\" (UID: \"0a5f3756-68b8-4e1d-97fc-75a9e6e6fb27\") " pod="openshift-ovn-kubernetes/ovnkube-node-vwtvz"
Sep 17 17:41:03.196548 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:03.196183    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/451c115d-30f1-4679-be41-ff6d2beedaf8-host\") pod \"tuned-vvjl9\" (UID: \"451c115d-30f1-4679-be41-ff6d2beedaf8\") " pod="openshift-cluster-node-tuning-operator/tuned-vvjl9"
Sep 17 17:41:03.196548 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:03.196217    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0a5f3756-68b8-4e1d-97fc-75a9e6e6fb27-node-log\") pod \"ovnkube-node-vwtvz\" (UID: \"0a5f3756-68b8-4e1d-97fc-75a9e6e6fb27\") " pod="openshift-ovn-kubernetes/ovnkube-node-vwtvz"
Sep 17 17:41:03.196548 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:03.196243    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e45ad616-0131-4f4e-adf4-b567b76edc38-serviceca\") pod \"node-ca-cfn7g\" (UID: \"e45ad616-0131-4f4e-adf4-b567b76edc38\") " pod="openshift-image-registry/node-ca-cfn7g"
Sep 17 17:41:03.196548 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:03.196268    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/7e56ce66-8c6f-4fa0-9560-6210a25593fa-device-dir\") pod \"aws-ebs-csi-driver-node-6xdld\" (UID: \"7e56ce66-8c6f-4fa0-9560-6210a25593fa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6xdld"
Sep 17 17:41:03.196548 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:03.196291    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0a5f3756-68b8-4e1d-97fc-75a9e6e6fb27-host-kubelet\") pod \"ovnkube-node-vwtvz\" (UID: \"0a5f3756-68b8-4e1d-97fc-75a9e6e6fb27\") " pod="openshift-ovn-kubernetes/ovnkube-node-vwtvz"
Sep 17 17:41:03.196548 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:03.196314    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0a5f3756-68b8-4e1d-97fc-75a9e6e6fb27-etc-openvswitch\") pod \"ovnkube-node-vwtvz\" (UID: \"0a5f3756-68b8-4e1d-97fc-75a9e6e6fb27\") " pod="openshift-ovn-kubernetes/ovnkube-node-vwtvz"
Sep 17 17:41:03.196548 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:03.196328    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/db73a8fe-d34c-41c8-8ba4-8040b4bacf13-dbus\") pod \"global-pull-secret-syncer-vjstm\" (UID: \"db73a8fe-d34c-41c8-8ba4-8040b4bacf13\") " pod="kube-system/global-pull-secret-syncer-vjstm"
Sep 17 17:41:03.196548 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:03.196348    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/451c115d-30f1-4679-be41-ff6d2beedaf8-etc-modprobe-d\") pod \"tuned-vvjl9\" (UID: \"451c115d-30f1-4679-be41-ff6d2beedaf8\") " pod="openshift-cluster-node-tuning-operator/tuned-vvjl9"
Sep 17 17:41:03.197280 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:03.196391    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0a5f3756-68b8-4e1d-97fc-75a9e6e6fb27-var-lib-openvswitch\") pod \"ovnkube-node-vwtvz\" (UID: \"0a5f3756-68b8-4e1d-97fc-75a9e6e6fb27\") " pod="openshift-ovn-kubernetes/ovnkube-node-vwtvz"
Sep 17 17:41:03.197280 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:03.196409    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/451c115d-30f1-4679-be41-ff6d2beedaf8-run\") pod \"tuned-vvjl9\" (UID: \"451c115d-30f1-4679-be41-ff6d2beedaf8\") " pod="openshift-cluster-node-tuning-operator/tuned-vvjl9"
Sep 17 17:41:03.197280 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:03.196426    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/451c115d-30f1-4679-be41-ff6d2beedaf8-lib-modules\") pod \"tuned-vvjl9\" (UID: \"451c115d-30f1-4679-be41-ff6d2beedaf8\") " pod="openshift-cluster-node-tuning-operator/tuned-vvjl9"
Sep 17 17:41:03.197280 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:03.196503    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/7e56ce66-8c6f-4fa0-9560-6210a25593fa-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6xdld\" (UID: \"7e56ce66-8c6f-4fa0-9560-6210a25593fa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6xdld"
Sep 17 17:41:03.197280 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:03.196585    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mnlzd\" (UniqueName: \"kubernetes.io/projected/5ab7aae3-053b-4e36-9c4a-c523f0cfb615-kube-api-access-mnlzd\") pod \"network-metrics-daemon-xg2wr\" (UID: \"5ab7aae3-053b-4e36-9c4a-c523f0cfb615\") " pod="openshift-multus/network-metrics-daemon-xg2wr"
Sep 17 17:41:03.197280 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:03.196611    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0a5f3756-68b8-4e1d-97fc-75a9e6e6fb27-host-run-netns\") pod \"ovnkube-node-vwtvz\" (UID: \"0a5f3756-68b8-4e1d-97fc-75a9e6e6fb27\") " pod="openshift-ovn-kubernetes/ovnkube-node-vwtvz"
Sep 17 17:41:03.197280 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:03.196670    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0a5f3756-68b8-4e1d-97fc-75a9e6e6fb27-host-cni-bin\") pod \"ovnkube-node-vwtvz\" (UID: \"0a5f3756-68b8-4e1d-97fc-75a9e6e6fb27\") " pod="openshift-ovn-kubernetes/ovnkube-node-vwtvz"
Sep 17 17:41:03.197280 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:03.196708    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/0a5f3756-68b8-4e1d-97fc-75a9e6e6fb27-ovnkube-config\") pod \"ovnkube-node-vwtvz\" (UID: \"0a5f3756-68b8-4e1d-97fc-75a9e6e6fb27\") " pod="openshift-ovn-kubernetes/ovnkube-node-vwtvz"
Sep 17 17:41:03.197666 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:03.197425    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-zrbgv"
Sep 17 17:41:03.199184 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:03.199163    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-2fqsl\""
Sep 17 17:41:03.199982 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:03.199962    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 17 17:41:03.200216 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:03.200196    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 17 17:41:03.200542 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:03.200482    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 17 17:41:03.200660 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:03.200642    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-j5jw8"
Sep 17 17:41:03.200845 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:03.200827    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 17 17:41:03.201425 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:03.201384    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-5sgb7\""
Sep 17 17:41:03.201642 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:03.201625    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 17 17:41:03.201642 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:03.201637    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 17 17:41:03.202111 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:03.202094    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 17 17:41:03.202510 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:03.202490    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-ptjgb"
Sep 17 17:41:03.203932 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:03.203439    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 17 17:41:03.204595 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:03.204575    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 17 17:41:03.204752 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:03.204736    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-v22fw\""
Sep 17 17:41:03.204876 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:03.204860    2567 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 17 17:41:03.205754 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:03.205734    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-dhfnq\""
Sep 17 17:41:03.205877 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:03.205855    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 17 17:41:03.206083 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:03.206062    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 17 17:41:03.229432 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:03.229409    2567 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-qbmbj"
Sep 17 17:41:03.240393 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:03.240355    2567 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-qbmbj"
Sep 17 17:41:03.286042 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:03.286021    2567 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 17 17:41:03.286498 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:03.286448    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-237.ec2.internal" event={"ID":"840f4093b21dc0fac911784fe7877e0e","Type":"ContainerStarted","Data":"4e2546499cbd4b2787c66a780c1f389feb4c38cdee7b5d6ff62f00888df7e95f"}
Sep 17 17:41:03.288058 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:03.288028    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-237.ec2.internal" event={"ID":"e678eb327b2639be46e3883082cf83ef","Type":"ContainerStarted","Data":"73d0a869b4d16f36a006af6e0cd7afecfa6f56933db3a9323727750d324aa71c"}
Sep 17 17:41:03.296904 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:03.296884    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/451c115d-30f1-4679-be41-ff6d2beedaf8-etc-sysconfig\") pod \"tuned-vvjl9\" (UID: \"451c115d-30f1-4679-be41-ff6d2beedaf8\") " pod="openshift-cluster-node-tuning-operator/tuned-vvjl9"
Sep 17 17:41:03.297006 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:03.296919    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mn68f\" (UniqueName: \"kubernetes.io/projected/bb03de28-d033-4586-8a80-54465decf01e-kube-api-access-mn68f\") pod \"multus-additional-cni-plugins-ptjgb\" (UID: \"bb03de28-d033-4586-8a80-54465decf01e\") " pod="openshift-multus/multus-additional-cni-plugins-ptjgb"
Sep 17 17:41:03.297006 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:03.296944    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/41a677ee-5541-4579-a6e9-93b915b89d2d-host-var-lib-cni-bin\") pod \"multus-dsxjr\" (UID: \"41a677ee-5541-4579-a6e9-93b915b89d2d\") " pod="openshift-multus/multus-dsxjr"
Sep 17 17:41:03.297006 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:03.296968    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nw6pc\" (UniqueName: \"kubernetes.io/projected/41a677ee-5541-4579-a6e9-93b915b89d2d-kube-api-access-nw6pc\") pod \"multus-dsxjr\" (UID: \"41a677ee-5541-4579-a6e9-93b915b89d2d\") " pod="openshift-multus/multus-dsxjr"
Sep 17 17:41:03.297006 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:03.296973    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/451c115d-30f1-4679-be41-ff6d2beedaf8-etc-sysconfig\") pod \"tuned-vvjl9\" (UID: \"451c115d-30f1-4679-be41-ff6d2beedaf8\") " pod="openshift-cluster-node-tuning-operator/tuned-vvjl9"
Sep 17 17:41:03.297006 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:03.296992    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kg27h\" (UniqueName: \"kubernetes.io/projected/0a5f3756-68b8-4e1d-97fc-75a9e6e6fb27-kube-api-access-kg27h\") pod \"ovnkube-node-vwtvz\" (UID: \"0a5f3756-68b8-4e1d-97fc-75a9e6e6fb27\") " pod="openshift-ovn-kubernetes/ovnkube-node-vwtvz"
Sep 17 17:41:03.297227 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:03.297030    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/41a677ee-5541-4579-a6e9-93b915b89d2d-system-cni-dir\") pod \"multus-dsxjr\" (UID: \"41a677ee-5541-4579-a6e9-93b915b89d2d\") " pod="openshift-multus/multus-dsxjr"
Sep 17 17:41:03.297227 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:03.297157    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/41a677ee-5541-4579-a6e9-93b915b89d2d-host-run-multus-certs\") pod \"multus-dsxjr\" (UID: \"41a677ee-5541-4579-a6e9-93b915b89d2d\") " pod="openshift-multus/multus-dsxjr"
Sep 17 17:41:03.297227 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:03.297194    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/41a677ee-5541-4579-a6e9-93b915b89d2d-etc-kubernetes\") pod \"multus-dsxjr\" (UID: \"41a677ee-5541-4579-a6e9-93b915b89d2d\") " pod="openshift-multus/multus-dsxjr"
Sep 17 17:41:03.297361 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:03.297229    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/451c115d-30f1-4679-be41-ff6d2beedaf8-sys\") pod \"tuned-vvjl9\" (UID: \"451c115d-30f1-4679-be41-ff6d2beedaf8\") " pod="openshift-cluster-node-tuning-operator/tuned-vvjl9"
Sep 17 17:41:03.297361 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:03.297254    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/451c115d-30f1-4679-be41-ff6d2beedaf8-tmp\") pod \"tuned-vvjl9\" (UID: \"451c115d-30f1-4679-be41-ff6d2beedaf8\") " pod="openshift-cluster-node-tuning-operator/tuned-vvjl9"
Sep 17 17:41:03.297361 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:03.297274    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bb03de28-d033-4586-8a80-54465decf01e-cnibin\") pod \"multus-additional-cni-plugins-ptjgb\" (UID: \"bb03de28-d033-4586-8a80-54465decf01e\") " pod="openshift-multus/multus-additional-cni-plugins-ptjgb"
Sep 17 17:41:03.297361 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:03.297289    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/41a677ee-5541-4579-a6e9-93b915b89d2d-host-run-netns\") pod \"multus-dsxjr\" (UID: \"41a677ee-5541-4579-a6e9-93b915b89d2d\") " pod="openshift-multus/multus-dsxjr"
Sep 17 17:41:03.297361 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:03.297304    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/0a5f3756-68b8-4e1d-97fc-75a9e6e6fb27-env-overrides\") pod \"ovnkube-node-vwtvz\" (UID: \"0a5f3756-68b8-4e1d-97fc-75a9e6e6fb27\") " pod="openshift-ovn-kubernetes/ovnkube-node-vwtvz"
Sep 17 17:41:03.297361 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:03.297332    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/bb03de28-d033-4586-8a80-54465decf01e-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-ptjgb\" (UID: \"bb03de28-d033-4586-8a80-54465decf01e\") " pod="openshift-multus/multus-additional-cni-plugins-ptjgb"
Sep 17 17:41:03.297361 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:03.297339    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/451c115d-30f1-4679-be41-ff6d2beedaf8-sys\") pod \"tuned-vvjl9\" (UID: \"451c115d-30f1-4679-be41-ff6d2beedaf8\") " pod="openshift-cluster-node-tuning-operator/tuned-vvjl9"
Sep 17 17:41:03.297743 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:03.297443    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e45ad616-0131-4f4e-adf4-b567b76edc38-serviceca\") pod \"node-ca-cfn7g\" (UID: \"e45ad616-0131-4f4e-adf4-b567b76edc38\") " pod="openshift-image-registry/node-ca-cfn7g"
Sep 17 17:41:03.297743 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:03.297474    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/451c115d-30f1-4679-be41-ff6d2beedaf8-etc-modprobe-d\") pod \"tuned-vvjl9\" (UID: \"451c115d-30f1-4679-be41-ff6d2beedaf8\") " pod="openshift-cluster-node-tuning-operator/tuned-vvjl9"
Sep 17 17:41:03.297743 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:03.297503    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/00f9603c-1639-4399-92ae-c1e4bf354395-iptables-alerter-script\") pod \"iptables-alerter-zrbgv\" (UID: \"00f9603c-1639-4399-92ae-c1e4bf354395\") " pod="openshift-network-operator/iptables-alerter-zrbgv"
Sep 17 17:41:03.297743 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:03.297542    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/451c115d-30f1-4679-be41-ff6d2beedaf8-lib-modules\") pod \"tuned-vvjl9\" (UID: \"451c115d-30f1-4679-be41-ff6d2beedaf8\") " pod="openshift-cluster-node-tuning-operator/tuned-vvjl9"
Sep 17 17:41:03.297743 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:03.297568    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0a5f3756-68b8-4e1d-97fc-75a9e6e6fb27-host-run-netns\") pod \"ovnkube-node-vwtvz\" (UID: \"0a5f3756-68b8-4e1d-97fc-75a9e6e6fb27\") " pod="openshift-ovn-kubernetes/ovnkube-node-vwtvz"
Sep 17 17:41:03.297743 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:03.297592    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0a5f3756-68b8-4e1d-97fc-75a9e6e6fb27-host-cni-bin\") pod \"ovnkube-node-vwtvz\" (UID: \"0a5f3756-68b8-4e1d-97fc-75a9e6e6fb27\") " pod="openshift-ovn-kubernetes/ovnkube-node-vwtvz"
Sep 17 17:41:03.297743 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:03.297609    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/0a5f3756-68b8-4e1d-97fc-75a9e6e6fb27-ovnkube-config\") pod \"ovnkube-node-vwtvz\" (UID: \"0a5f3756-68b8-4e1d-97fc-75a9e6e6fb27\") " pod="openshift-ovn-kubernetes/ovnkube-node-vwtvz"
Sep 17 17:41:03.297743 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:03.297623    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/db73a8fe-d34c-41c8-8ba4-8040b4bacf13-original-pull-secret\") pod \"global-pull-secret-syncer-vjstm\" (UID: \"db73a8fe-d34c-41c8-8ba4-8040b4bacf13\") " pod="kube-system/global-pull-secret-syncer-vjstm"
Sep 17 17:41:03.297743 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:03.297640    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/451c115d-30f1-4679-be41-ff6d2beedaf8-etc-sysctl-conf\") pod \"tuned-vvjl9\" (UID: \"451c115d-30f1-4679-be41-ff6d2beedaf8\") " pod="openshift-cluster-node-tuning-operator/tuned-vvjl9"
Sep 17 17:41:03.297743 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:03.297658    2567 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 17 17:41:03.297743 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:03.297707    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/7e56ce66-8c6f-4fa0-9560-6210a25593fa-sys-fs\") pod \"aws-ebs-csi-driver-node-6xdld\" (UID: \"7e56ce66-8c6f-4fa0-9560-6210a25593fa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6xdld"
Sep 17 17:41:03.298234 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:03.297793    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/451c115d-30f1-4679-be41-ff6d2beedaf8-etc-modprobe-d\") pod \"tuned-vvjl9\" (UID: \"451c115d-30f1-4679-be41-ff6d2beedaf8\") " pod="openshift-cluster-node-tuning-operator/tuned-vvjl9"
Sep 17 17:41:03.298234 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:03.297852    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/0a5f3756-68b8-4e1d-97fc-75a9e6e6fb27-env-overrides\") pod \"ovnkube-node-vwtvz\" (UID: \"0a5f3756-68b8-4e1d-97fc-75a9e6e6fb27\") " pod="openshift-ovn-kubernetes/ovnkube-node-vwtvz"
Sep 17 17:41:03.298234 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:03.297878    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0a5f3756-68b8-4e1d-97fc-75a9e6e6fb27-host-run-netns\") pod \"ovnkube-node-vwtvz\" (UID: \"0a5f3756-68b8-4e1d-97fc-75a9e6e6fb27\") " pod="openshift-ovn-kubernetes/ovnkube-node-vwtvz"
Sep 17 17:41:03.298234 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:03.297857    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/451c115d-30f1-4679-be41-ff6d2beedaf8-lib-modules\") pod \"tuned-vvjl9\" (UID: \"451c115d-30f1-4679-be41-ff6d2beedaf8\") " pod="openshift-cluster-node-tuning-operator/tuned-vvjl9"
Sep 17 17:41:03.298234 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:03.297913    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0a5f3756-68b8-4e1d-97fc-75a9e6e6fb27-host-cni-bin\") pod \"ovnkube-node-vwtvz\" (UID: \"0a5f3756-68b8-4e1d-97fc-75a9e6e6fb27\") " pod="openshift-ovn-kubernetes/ovnkube-node-vwtvz"
Sep 17 17:41:03.298234 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:03.297927    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e45ad616-0131-4f4e-adf4-b567b76edc38-serviceca\") pod \"node-ca-cfn7g\" (UID: \"e45ad616-0131-4f4e-adf4-b567b76edc38\") " pod="openshift-image-registry/node-ca-cfn7g"
Sep 17 17:41:03.298234 ip-10-0-139-237 kubenswrapper[2567]: E0917 17:41:03.297998    2567 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 17:41:03.298234 ip-10-0-139-237 kubenswrapper[2567]: E0917 17:41:03.298054    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/db73a8fe-d34c-41c8-8ba4-8040b4bacf13-original-pull-secret podName:db73a8fe-d34c-41c8-8ba4-8040b4bacf13 nodeName:}" failed. No retries permitted until 2026-09-17 17:41:03.798034944 +0000 UTC m=+4.821640079 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/db73a8fe-d34c-41c8-8ba4-8040b4bacf13-original-pull-secret") pod "global-pull-secret-syncer-vjstm" (UID: "db73a8fe-d34c-41c8-8ba4-8040b4bacf13") : object "kube-system"/"original-pull-secret" not registered
Sep 17 17:41:03.298234 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:03.298070    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/451c115d-30f1-4679-be41-ff6d2beedaf8-etc-sysctl-conf\") pod \"tuned-vvjl9\" (UID: \"451c115d-30f1-4679-be41-ff6d2beedaf8\") " pod="openshift-cluster-node-tuning-operator/tuned-vvjl9"
Sep 17 17:41:03.298234 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:03.297673    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/7e56ce66-8c6f-4fa0-9560-6210a25593fa-sys-fs\") pod \"aws-ebs-csi-driver-node-6xdld\" (UID: \"7e56ce66-8c6f-4fa0-9560-6210a25593fa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6xdld"
Sep 17 17:41:03.298234 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:03.298207    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5ab7aae3-053b-4e36-9c4a-c523f0cfb615-metrics-certs\") pod \"network-metrics-daemon-xg2wr\" (UID: \"5ab7aae3-053b-4e36-9c4a-c523f0cfb615\") " pod="openshift-multus/network-metrics-daemon-xg2wr"
Sep 17 17:41:03.298234 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:03.298234    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0a5f3756-68b8-4e1d-97fc-75a9e6e6fb27-run-openvswitch\") pod \"ovnkube-node-vwtvz\" (UID: \"0a5f3756-68b8-4e1d-97fc-75a9e6e6fb27\") " pod="openshift-ovn-kubernetes/ovnkube-node-vwtvz"
Sep 17 17:41:03.298840 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:03.298257    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/db73a8fe-d34c-41c8-8ba4-8040b4bacf13-kubelet-config\") pod \"global-pull-secret-syncer-vjstm\" (UID: \"db73a8fe-d34c-41c8-8ba4-8040b4bacf13\") " pod="kube-system/global-pull-secret-syncer-vjstm"
Sep 17 17:41:03.298840 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:03.298280    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/451c115d-30f1-4679-be41-ff6d2beedaf8-etc-sysctl-d\") pod \"tuned-vvjl9\" (UID: \"451c115d-30f1-4679-be41-ff6d2beedaf8\") " pod="openshift-cluster-node-tuning-operator/tuned-vvjl9"
Sep 17 17:41:03.298840 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:03.298303    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/41a677ee-5541-4579-a6e9-93b915b89d2d-hostroot\") pod \"multus-dsxjr\" (UID: \"41a677ee-5541-4579-a6e9-93b915b89d2d\") " pod="openshift-multus/multus-dsxjr"
Sep 17 17:41:03.298840 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:03.298320    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xbxgz\" (UniqueName: \"kubernetes.io/projected/e45ad616-0131-4f4e-adf4-b567b76edc38-kube-api-access-xbxgz\") pod \"node-ca-cfn7g\" (UID: \"e45ad616-0131-4f4e-adf4-b567b76edc38\") " pod="openshift-image-registry/node-ca-cfn7g"
Sep 17 17:41:03.298840 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:03.298336    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/7e56ce66-8c6f-4fa0-9560-6210a25593fa-socket-dir\") pod \"aws-ebs-csi-driver-node-6xdld\" (UID: \"7e56ce66-8c6f-4fa0-9560-6210a25593fa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6xdld"
Sep 17 17:41:03.298840 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:03.298363    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/0a5f3756-68b8-4e1d-97fc-75a9e6e6fb27-ovnkube-config\") pod \"ovnkube-node-vwtvz\" (UID: \"0a5f3756-68b8-4e1d-97fc-75a9e6e6fb27\") " pod="openshift-ovn-kubernetes/ovnkube-node-vwtvz"
Sep 17 17:41:03.298840 ip-10-0-139-237 kubenswrapper[2567]: E0917 17:41:03.298382    2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:41:03.298840 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:03.298414    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/db73a8fe-d34c-41c8-8ba4-8040b4bacf13-kubelet-config\") pod \"global-pull-secret-syncer-vjstm\" (UID: \"db73a8fe-d34c-41c8-8ba4-8040b4bacf13\") " pod="kube-system/global-pull-secret-syncer-vjstm"
Sep 17 17:41:03.298840 ip-10-0-139-237 kubenswrapper[2567]: E0917 17:41:03.298427    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5ab7aae3-053b-4e36-9c4a-c523f0cfb615-metrics-certs podName:5ab7aae3-053b-4e36-9c4a-c523f0cfb615 nodeName:}" failed. No retries permitted until 2026-09-17 17:41:03.798412414 +0000 UTC m=+4.822017546 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5ab7aae3-053b-4e36-9c4a-c523f0cfb615-metrics-certs") pod "network-metrics-daemon-xg2wr" (UID: "5ab7aae3-053b-4e36-9c4a-c523f0cfb615") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:41:03.298840 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:03.298359    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/7e56ce66-8c6f-4fa0-9560-6210a25593fa-etc-selinux\") pod \"aws-ebs-csi-driver-node-6xdld\" (UID: \"7e56ce66-8c6f-4fa0-9560-6210a25593fa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6xdld"
Sep 17 17:41:03.298840 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:03.298469    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gb66b\" (UniqueName: \"kubernetes.io/projected/7e56ce66-8c6f-4fa0-9560-6210a25593fa-kube-api-access-gb66b\") pod \"aws-ebs-csi-driver-node-6xdld\" (UID: \"7e56ce66-8c6f-4fa0-9560-6210a25593fa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6xdld"
Sep 17 17:41:03.298840 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:03.298472    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/7e56ce66-8c6f-4fa0-9560-6210a25593fa-etc-selinux\") pod \"aws-ebs-csi-driver-node-6xdld\" (UID: \"7e56ce66-8c6f-4fa0-9560-6210a25593fa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6xdld"
Sep 17 17:41:03.298840 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:03.298502    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/451c115d-30f1-4679-be41-ff6d2beedaf8-etc-sysctl-d\") pod \"tuned-vvjl9\" (UID: \"451c115d-30f1-4679-be41-ff6d2beedaf8\") " pod="openshift-cluster-node-tuning-operator/tuned-vvjl9"
Sep 17 17:41:03.298840 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:03.298506    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0a5f3756-68b8-4e1d-97fc-75a9e6e6fb27-run-systemd\") pod \"ovnkube-node-vwtvz\" (UID: \"0a5f3756-68b8-4e1d-97fc-75a9e6e6fb27\") " pod="openshift-ovn-kubernetes/ovnkube-node-vwtvz"
Sep 17 17:41:03.298840 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:03.298549    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0a5f3756-68b8-4e1d-97fc-75a9e6e6fb27-host-cni-netd\") pod \"ovnkube-node-vwtvz\" (UID: \"0a5f3756-68b8-4e1d-97fc-75a9e6e6fb27\") " pod="openshift-ovn-kubernetes/ovnkube-node-vwtvz"
Sep 17 17:41:03.298840 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:03.298572    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/451c115d-30f1-4679-be41-ff6d2beedaf8-var-lib-kubelet\") pod \"tuned-vvjl9\" (UID: \"451c115d-30f1-4679-be41-ff6d2beedaf8\") " pod="openshift-cluster-node-tuning-operator/tuned-vvjl9"
Sep 17 17:41:03.298840 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:03.298596    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0a5f3756-68b8-4e1d-97fc-75a9e6e6fb27-run-ovn\") pod \"ovnkube-node-vwtvz\" (UID: \"0a5f3756-68b8-4e1d-97fc-75a9e6e6fb27\") " pod="openshift-ovn-kubernetes/ovnkube-node-vwtvz"
Sep 17 17:41:03.299667 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:03.298619    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0a5f3756-68b8-4e1d-97fc-75a9e6e6fb27-log-socket\") pod \"ovnkube-node-vwtvz\" (UID: \"0a5f3756-68b8-4e1d-97fc-75a9e6e6fb27\") " pod="openshift-ovn-kubernetes/ovnkube-node-vwtvz"
Sep 17 17:41:03.299667 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:03.298316    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0a5f3756-68b8-4e1d-97fc-75a9e6e6fb27-run-openvswitch\") pod \"ovnkube-node-vwtvz\" (UID: \"0a5f3756-68b8-4e1d-97fc-75a9e6e6fb27\") " pod="openshift-ovn-kubernetes/ovnkube-node-vwtvz"
Sep 17 17:41:03.299667 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:03.298644    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/7e56ce66-8c6f-4fa0-9560-6210a25593fa-registration-dir\") pod \"aws-ebs-csi-driver-node-6xdld\" (UID: \"7e56ce66-8c6f-4fa0-9560-6210a25593fa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6xdld"
Sep 17 17:41:03.299667 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:03.298669    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0a5f3756-68b8-4e1d-97fc-75a9e6e6fb27-systemd-units\") pod \"ovnkube-node-vwtvz\" (UID: \"0a5f3756-68b8-4e1d-97fc-75a9e6e6fb27\") " pod="openshift-ovn-kubernetes/ovnkube-node-vwtvz"
Sep 17 17:41:03.299667 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:03.298703    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/9a0f326b-5e07-4829-bc1f-084edb7f3b17-konnectivity-ca\") pod \"konnectivity-agent-j5jw8\" (UID: \"9a0f326b-5e07-4829-bc1f-084edb7f3b17\") " pod="kube-system/konnectivity-agent-j5jw8"
Sep 17 17:41:03.299667 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:03.298708    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/7e56ce66-8c6f-4fa0-9560-6210a25593fa-socket-dir\") pod \"aws-ebs-csi-driver-node-6xdld\" (UID: \"7e56ce66-8c6f-4fa0-9560-6210a25593fa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6xdld"
Sep 17 17:41:03.299667 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:03.298739    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/9a0f326b-5e07-4829-bc1f-084edb7f3b17-agent-certs\") pod \"konnectivity-agent-j5jw8\" (UID: \"9a0f326b-5e07-4829-bc1f-084edb7f3b17\") " pod="kube-system/konnectivity-agent-j5jw8"
Sep 17 17:41:03.299667 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:03.298746    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0a5f3756-68b8-4e1d-97fc-75a9e6e6fb27-host-cni-netd\") pod \"ovnkube-node-vwtvz\" (UID: \"0a5f3756-68b8-4e1d-97fc-75a9e6e6fb27\") " pod="openshift-ovn-kubernetes/ovnkube-node-vwtvz"
Sep 17 17:41:03.299667 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:03.298766    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0a5f3756-68b8-4e1d-97fc-75a9e6e6fb27-run-ovn\") pod \"ovnkube-node-vwtvz\" (UID: \"0a5f3756-68b8-4e1d-97fc-75a9e6e6fb27\") " pod="openshift-ovn-kubernetes/ovnkube-node-vwtvz"
Sep 17 17:41:03.299667 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:03.298785    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0a5f3756-68b8-4e1d-97fc-75a9e6e6fb27-run-systemd\") pod \"ovnkube-node-vwtvz\" (UID: \"0a5f3756-68b8-4e1d-97fc-75a9e6e6fb27\") " pod="openshift-ovn-kubernetes/ovnkube-node-vwtvz"
Sep 17 17:41:03.299667 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:03.298768    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/19204c4a-e05d-4159-94c8-477036fa461a-tmp-dir\") pod \"node-resolver-454w2\" (UID: \"19204c4a-e05d-4159-94c8-477036fa461a\") " pod="openshift-dns/node-resolver-454w2"
Sep 17 17:41:03.299667 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:03.298811    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0a5f3756-68b8-4e1d-97fc-75a9e6e6fb27-log-socket\") pod \"ovnkube-node-vwtvz\" (UID: \"0a5f3756-68b8-4e1d-97fc-75a9e6e6fb27\") " pod="openshift-ovn-kubernetes/ovnkube-node-vwtvz"
Sep 17 17:41:03.299667 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:03.298821    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e45ad616-0131-4f4e-adf4-b567b76edc38-host\") pod \"node-ca-cfn7g\" (UID: \"e45ad616-0131-4f4e-adf4-b567b76edc38\") " pod="openshift-image-registry/node-ca-cfn7g"
Sep 17 17:41:03.299667 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:03.298827    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/451c115d-30f1-4679-be41-ff6d2beedaf8-var-lib-kubelet\") pod \"tuned-vvjl9\" (UID: \"451c115d-30f1-4679-be41-ff6d2beedaf8\") " pod="openshift-cluster-node-tuning-operator/tuned-vvjl9"
Sep 17 17:41:03.299667 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:03.298850    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0a5f3756-68b8-4e1d-97fc-75a9e6e6fb27-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-vwtvz\" (UID: \"0a5f3756-68b8-4e1d-97fc-75a9e6e6fb27\") " pod="openshift-ovn-kubernetes/ovnkube-node-vwtvz"
Sep 17 17:41:03.299667 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:03.298859    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/7e56ce66-8c6f-4fa0-9560-6210a25593fa-registration-dir\") pod \"aws-ebs-csi-driver-node-6xdld\" (UID: \"7e56ce66-8c6f-4fa0-9560-6210a25593fa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6xdld"
Sep 17 17:41:03.299667 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:03.298863    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e45ad616-0131-4f4e-adf4-b567b76edc38-host\") pod \"node-ca-cfn7g\" (UID: \"e45ad616-0131-4f4e-adf4-b567b76edc38\") " pod="openshift-image-registry/node-ca-cfn7g"
Sep 17 17:41:03.300346 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:03.298874    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/0a5f3756-68b8-4e1d-97fc-75a9e6e6fb27-ovnkube-script-lib\") pod \"ovnkube-node-vwtvz\" (UID: \"0a5f3756-68b8-4e1d-97fc-75a9e6e6fb27\") " pod="openshift-ovn-kubernetes/ovnkube-node-vwtvz"
Sep 17 17:41:03.300346 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:03.298903    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/41a677ee-5541-4579-a6e9-93b915b89d2d-cni-binary-copy\") pod \"multus-dsxjr\" (UID: \"41a677ee-5541-4579-a6e9-93b915b89d2d\") " pod="openshift-multus/multus-dsxjr"
Sep 17 17:41:03.300346 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:03.298932    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0a5f3756-68b8-4e1d-97fc-75a9e6e6fb27-systemd-units\") pod \"ovnkube-node-vwtvz\" (UID: \"0a5f3756-68b8-4e1d-97fc-75a9e6e6fb27\") " pod="openshift-ovn-kubernetes/ovnkube-node-vwtvz"
Sep 17 17:41:03.300346 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:03.298937    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/41a677ee-5541-4579-a6e9-93b915b89d2d-host-run-k8s-cni-cncf-io\") pod \"multus-dsxjr\" (UID: \"41a677ee-5541-4579-a6e9-93b915b89d2d\") " pod="openshift-multus/multus-dsxjr"
Sep 17 17:41:03.300346 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:03.298903    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0a5f3756-68b8-4e1d-97fc-75a9e6e6fb27-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-vwtvz\" (UID: \"0a5f3756-68b8-4e1d-97fc-75a9e6e6fb27\") " pod="openshift-ovn-kubernetes/ovnkube-node-vwtvz"
Sep 17 17:41:03.300346 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:03.298963    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/41a677ee-5541-4579-a6e9-93b915b89d2d-host-var-lib-kubelet\") pod \"multus-dsxjr\" (UID: \"41a677ee-5541-4579-a6e9-93b915b89d2d\") " pod="openshift-multus/multus-dsxjr"
Sep 17 17:41:03.300346 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:03.298991    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j8wst\" (UniqueName: \"kubernetes.io/projected/00f9603c-1639-4399-92ae-c1e4bf354395-kube-api-access-j8wst\") pod \"iptables-alerter-zrbgv\" (UID: \"00f9603c-1639-4399-92ae-c1e4bf354395\") " pod="openshift-network-operator/iptables-alerter-zrbgv"
Sep 17 17:41:03.300346 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:03.299016    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0a5f3756-68b8-4e1d-97fc-75a9e6e6fb27-host-run-ovn-kubernetes\") pod \"ovnkube-node-vwtvz\" (UID: \"0a5f3756-68b8-4e1d-97fc-75a9e6e6fb27\") " pod="openshift-ovn-kubernetes/ovnkube-node-vwtvz"
Sep 17 17:41:03.300346 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:03.299040    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/451c115d-30f1-4679-be41-ff6d2beedaf8-host\") pod \"tuned-vvjl9\" (UID: \"451c115d-30f1-4679-be41-ff6d2beedaf8\") " pod="openshift-cluster-node-tuning-operator/tuned-vvjl9"
Sep 17 17:41:03.300346 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:03.299074    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bb03de28-d033-4586-8a80-54465decf01e-system-cni-dir\") pod \"multus-additional-cni-plugins-ptjgb\" (UID: \"bb03de28-d033-4586-8a80-54465decf01e\") " pod="openshift-multus/multus-additional-cni-plugins-ptjgb"
Sep 17 17:41:03.300346 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:03.299095    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/451c115d-30f1-4679-be41-ff6d2beedaf8-host\") pod \"tuned-vvjl9\" (UID: \"451c115d-30f1-4679-be41-ff6d2beedaf8\") " pod="openshift-cluster-node-tuning-operator/tuned-vvjl9"
Sep 17 17:41:03.300346 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:03.299133    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/41a677ee-5541-4579-a6e9-93b915b89d2d-cnibin\") pod \"multus-dsxjr\" (UID: \"41a677ee-5541-4579-a6e9-93b915b89d2d\") " pod="openshift-multus/multus-dsxjr"
Sep 17 17:41:03.300346 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:03.299142    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0a5f3756-68b8-4e1d-97fc-75a9e6e6fb27-host-run-ovn-kubernetes\") pod \"ovnkube-node-vwtvz\" (UID: \"0a5f3756-68b8-4e1d-97fc-75a9e6e6fb27\") " pod="openshift-ovn-kubernetes/ovnkube-node-vwtvz"
Sep 17 17:41:03.300346 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:03.299158    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/41a677ee-5541-4579-a6e9-93b915b89d2d-multus-socket-dir-parent\") pod \"multus-dsxjr\" (UID: \"41a677ee-5541-4579-a6e9-93b915b89d2d\") " pod="openshift-multus/multus-dsxjr"
Sep 17 17:41:03.300346 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:03.299183    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/41a677ee-5541-4579-a6e9-93b915b89d2d-host-var-lib-cni-multus\") pod \"multus-dsxjr\" (UID: \"41a677ee-5541-4579-a6e9-93b915b89d2d\") " pod="openshift-multus/multus-dsxjr"
Sep 17 17:41:03.300346 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:03.299201    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/00f9603c-1639-4399-92ae-c1e4bf354395-host-slash\") pod \"iptables-alerter-zrbgv\" (UID: \"00f9603c-1639-4399-92ae-c1e4bf354395\") " pod="openshift-network-operator/iptables-alerter-zrbgv"
Sep 17 17:41:03.300346 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:03.299221    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0a5f3756-68b8-4e1d-97fc-75a9e6e6fb27-node-log\") pod \"ovnkube-node-vwtvz\" (UID: \"0a5f3756-68b8-4e1d-97fc-75a9e6e6fb27\") " pod="openshift-ovn-kubernetes/ovnkube-node-vwtvz"
Sep 17 17:41:03.301140 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:03.299279    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0a5f3756-68b8-4e1d-97fc-75a9e6e6fb27-node-log\") pod \"ovnkube-node-vwtvz\" (UID: \"0a5f3756-68b8-4e1d-97fc-75a9e6e6fb27\") " pod="openshift-ovn-kubernetes/ovnkube-node-vwtvz"
Sep 17 17:41:03.301140 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:03.299293    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/7e56ce66-8c6f-4fa0-9560-6210a25593fa-device-dir\") pod \"aws-ebs-csi-driver-node-6xdld\" (UID: \"7e56ce66-8c6f-4fa0-9560-6210a25593fa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6xdld"
Sep 17 17:41:03.301140 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:03.299328    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0a5f3756-68b8-4e1d-97fc-75a9e6e6fb27-host-kubelet\") pod \"ovnkube-node-vwtvz\" (UID: \"0a5f3756-68b8-4e1d-97fc-75a9e6e6fb27\") " pod="openshift-ovn-kubernetes/ovnkube-node-vwtvz"
Sep 17 17:41:03.301140 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:03.299350    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0a5f3756-68b8-4e1d-97fc-75a9e6e6fb27-etc-openvswitch\") pod \"ovnkube-node-vwtvz\" (UID: \"0a5f3756-68b8-4e1d-97fc-75a9e6e6fb27\") " pod="openshift-ovn-kubernetes/ovnkube-node-vwtvz"
Sep 17 17:41:03.301140 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:03.299364    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/0a5f3756-68b8-4e1d-97fc-75a9e6e6fb27-ovnkube-script-lib\") pod \"ovnkube-node-vwtvz\" (UID: \"0a5f3756-68b8-4e1d-97fc-75a9e6e6fb27\") " pod="openshift-ovn-kubernetes/ovnkube-node-vwtvz"
Sep 17 17:41:03.301140 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:03.299375    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/db73a8fe-d34c-41c8-8ba4-8040b4bacf13-dbus\") pod \"global-pull-secret-syncer-vjstm\" (UID: \"db73a8fe-d34c-41c8-8ba4-8040b4bacf13\") " pod="kube-system/global-pull-secret-syncer-vjstm"
Sep 17 17:41:03.301140 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:03.299399    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/41a677ee-5541-4579-a6e9-93b915b89d2d-multus-conf-dir\") pod \"multus-dsxjr\" (UID: \"41a677ee-5541-4579-a6e9-93b915b89d2d\") " pod="openshift-multus/multus-dsxjr"
Sep 17 17:41:03.301140 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:03.299406    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0a5f3756-68b8-4e1d-97fc-75a9e6e6fb27-host-kubelet\") pod \"ovnkube-node-vwtvz\" (UID: \"0a5f3756-68b8-4e1d-97fc-75a9e6e6fb27\") " pod="openshift-ovn-kubernetes/ovnkube-node-vwtvz"
Sep 17 17:41:03.301140 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:03.299423    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/41a677ee-5541-4579-a6e9-93b915b89d2d-multus-daemon-config\") pod \"multus-dsxjr\" (UID: \"41a677ee-5541-4579-a6e9-93b915b89d2d\") " pod="openshift-multus/multus-dsxjr"
Sep 17 17:41:03.301140 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:03.299433    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/7e56ce66-8c6f-4fa0-9560-6210a25593fa-device-dir\") pod \"aws-ebs-csi-driver-node-6xdld\" (UID: \"7e56ce66-8c6f-4fa0-9560-6210a25593fa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6xdld"
Sep 17 17:41:03.301140 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:03.299447    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0a5f3756-68b8-4e1d-97fc-75a9e6e6fb27-var-lib-openvswitch\") pod \"ovnkube-node-vwtvz\" (UID: \"0a5f3756-68b8-4e1d-97fc-75a9e6e6fb27\") " pod="openshift-ovn-kubernetes/ovnkube-node-vwtvz"
Sep 17 17:41:03.301140 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:03.299425    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0a5f3756-68b8-4e1d-97fc-75a9e6e6fb27-etc-openvswitch\") pod \"ovnkube-node-vwtvz\" (UID: \"0a5f3756-68b8-4e1d-97fc-75a9e6e6fb27\") " pod="openshift-ovn-kubernetes/ovnkube-node-vwtvz"
Sep 17 17:41:03.301140 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:03.299473    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/451c115d-30f1-4679-be41-ff6d2beedaf8-run\") pod \"tuned-vvjl9\" (UID: \"451c115d-30f1-4679-be41-ff6d2beedaf8\") " pod="openshift-cluster-node-tuning-operator/tuned-vvjl9"
Sep 17 17:41:03.301140 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:03.299496    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lndx7\" (UniqueName: \"kubernetes.io/projected/6d0a7973-cc86-4fda-9056-11b5923bd14c-kube-api-access-lndx7\") pod \"network-check-target-4cb4f\" (UID: \"6d0a7973-cc86-4fda-9056-11b5923bd14c\") " pod="openshift-network-diagnostics/network-check-target-4cb4f"
Sep 17 17:41:03.301140 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:03.299502    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0a5f3756-68b8-4e1d-97fc-75a9e6e6fb27-var-lib-openvswitch\") pod \"ovnkube-node-vwtvz\" (UID: \"0a5f3756-68b8-4e1d-97fc-75a9e6e6fb27\") " pod="openshift-ovn-kubernetes/ovnkube-node-vwtvz"
Sep 17 17:41:03.301140 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:03.299524    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bb03de28-d033-4586-8a80-54465decf01e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-ptjgb\" (UID: \"bb03de28-d033-4586-8a80-54465decf01e\") " pod="openshift-multus/multus-additional-cni-plugins-ptjgb"
Sep 17 17:41:03.301140 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:03.299588    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/db73a8fe-d34c-41c8-8ba4-8040b4bacf13-dbus\") pod \"global-pull-secret-syncer-vjstm\" (UID: \"db73a8fe-d34c-41c8-8ba4-8040b4bacf13\") " pod="kube-system/global-pull-secret-syncer-vjstm"
Sep 17 17:41:03.301683 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:03.299595    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/bb03de28-d033-4586-8a80-54465decf01e-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-ptjgb\" (UID: \"bb03de28-d033-4586-8a80-54465decf01e\") " pod="openshift-multus/multus-additional-cni-plugins-ptjgb"
Sep 17 17:41:03.301683 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:03.299615    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/451c115d-30f1-4679-be41-ff6d2beedaf8-run\") pod \"tuned-vvjl9\" (UID: \"451c115d-30f1-4679-be41-ff6d2beedaf8\") " pod="openshift-cluster-node-tuning-operator/tuned-vvjl9"
Sep 17 17:41:03.301683 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:03.299628    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/7e56ce66-8c6f-4fa0-9560-6210a25593fa-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6xdld\" (UID: \"7e56ce66-8c6f-4fa0-9560-6210a25593fa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6xdld"
Sep 17 17:41:03.301683 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:03.299657    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mnlzd\" (UniqueName: \"kubernetes.io/projected/5ab7aae3-053b-4e36-9c4a-c523f0cfb615-kube-api-access-mnlzd\") pod \"network-metrics-daemon-xg2wr\" (UID: \"5ab7aae3-053b-4e36-9c4a-c523f0cfb615\") " pod="openshift-multus/network-metrics-daemon-xg2wr"
Sep 17 17:41:03.301683 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:03.299667    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/7e56ce66-8c6f-4fa0-9560-6210a25593fa-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6xdld\" (UID: \"7e56ce66-8c6f-4fa0-9560-6210a25593fa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6xdld"
Sep 17 17:41:03.301683 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:03.299681    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hrmln\" (UniqueName: \"kubernetes.io/projected/19204c4a-e05d-4159-94c8-477036fa461a-kube-api-access-hrmln\") pod \"node-resolver-454w2\" (UID: \"19204c4a-e05d-4159-94c8-477036fa461a\") " pod="openshift-dns/node-resolver-454w2"
Sep 17 17:41:03.301683 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:03.299706    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/0a5f3756-68b8-4e1d-97fc-75a9e6e6fb27-ovn-node-metrics-cert\") pod \"ovnkube-node-vwtvz\" (UID: \"0a5f3756-68b8-4e1d-97fc-75a9e6e6fb27\") " pod="openshift-ovn-kubernetes/ovnkube-node-vwtvz"
Sep 17 17:41:03.301683 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:03.299731    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/451c115d-30f1-4679-be41-ff6d2beedaf8-etc-kubernetes\") pod \"tuned-vvjl9\" (UID: \"451c115d-30f1-4679-be41-ff6d2beedaf8\") " pod="openshift-cluster-node-tuning-operator/tuned-vvjl9"
Sep 17 17:41:03.301683 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:03.299762    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/451c115d-30f1-4679-be41-ff6d2beedaf8-etc-systemd\") pod \"tuned-vvjl9\" (UID: \"451c115d-30f1-4679-be41-ff6d2beedaf8\") " pod="openshift-cluster-node-tuning-operator/tuned-vvjl9"
Sep 17 17:41:03.301683 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:03.299790    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/451c115d-30f1-4679-be41-ff6d2beedaf8-etc-tuned\") pod \"tuned-vvjl9\" (UID: \"451c115d-30f1-4679-be41-ff6d2beedaf8\") " pod="openshift-cluster-node-tuning-operator/tuned-vvjl9"
Sep 17 17:41:03.301683 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:03.299818    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mmpqq\" (UniqueName: \"kubernetes.io/projected/451c115d-30f1-4679-be41-ff6d2beedaf8-kube-api-access-mmpqq\") pod \"tuned-vvjl9\" (UID: \"451c115d-30f1-4679-be41-ff6d2beedaf8\") " pod="openshift-cluster-node-tuning-operator/tuned-vvjl9"
Sep 17 17:41:03.301683 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:03.299846    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bb03de28-d033-4586-8a80-54465decf01e-cni-binary-copy\") pod \"multus-additional-cni-plugins-ptjgb\" (UID: \"bb03de28-d033-4586-8a80-54465decf01e\") " pod="openshift-multus/multus-additional-cni-plugins-ptjgb"
Sep 17 17:41:03.301683 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:03.299859    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/451c115d-30f1-4679-be41-ff6d2beedaf8-etc-systemd\") pod \"tuned-vvjl9\" (UID: \"451c115d-30f1-4679-be41-ff6d2beedaf8\") " pod="openshift-cluster-node-tuning-operator/tuned-vvjl9"
Sep 17 17:41:03.301683 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:03.299872    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/19204c4a-e05d-4159-94c8-477036fa461a-hosts-file\") pod \"node-resolver-454w2\" (UID: \"19204c4a-e05d-4159-94c8-477036fa461a\") " pod="openshift-dns/node-resolver-454w2"
Sep 17 17:41:03.301683 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:03.299904    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/41a677ee-5541-4579-a6e9-93b915b89d2d-multus-cni-dir\") pod \"multus-dsxjr\" (UID: \"41a677ee-5541-4579-a6e9-93b915b89d2d\") " pod="openshift-multus/multus-dsxjr"
Sep 17 17:41:03.301683 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:03.299931    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0a5f3756-68b8-4e1d-97fc-75a9e6e6fb27-host-slash\") pod \"ovnkube-node-vwtvz\" (UID: \"0a5f3756-68b8-4e1d-97fc-75a9e6e6fb27\") " pod="openshift-ovn-kubernetes/ovnkube-node-vwtvz"
Sep 17 17:41:03.301683 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:03.300013    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0a5f3756-68b8-4e1d-97fc-75a9e6e6fb27-host-slash\") pod \"ovnkube-node-vwtvz\" (UID: \"0a5f3756-68b8-4e1d-97fc-75a9e6e6fb27\") " pod="openshift-ovn-kubernetes/ovnkube-node-vwtvz"
Sep 17 17:41:03.302380 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:03.300070    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/451c115d-30f1-4679-be41-ff6d2beedaf8-etc-kubernetes\") pod \"tuned-vvjl9\" (UID: \"451c115d-30f1-4679-be41-ff6d2beedaf8\") " pod="openshift-cluster-node-tuning-operator/tuned-vvjl9"
Sep 17 17:41:03.302380 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:03.301472    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/451c115d-30f1-4679-be41-ff6d2beedaf8-tmp\") pod \"tuned-vvjl9\" (UID: \"451c115d-30f1-4679-be41-ff6d2beedaf8\") " pod="openshift-cluster-node-tuning-operator/tuned-vvjl9"
Sep 17 17:41:03.302380 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:03.302135    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/0a5f3756-68b8-4e1d-97fc-75a9e6e6fb27-ovn-node-metrics-cert\") pod \"ovnkube-node-vwtvz\" (UID: \"0a5f3756-68b8-4e1d-97fc-75a9e6e6fb27\") " pod="openshift-ovn-kubernetes/ovnkube-node-vwtvz"
Sep 17 17:41:03.304451 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:03.304418    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/451c115d-30f1-4679-be41-ff6d2beedaf8-etc-tuned\") pod \"tuned-vvjl9\" (UID: \"451c115d-30f1-4679-be41-ff6d2beedaf8\") " pod="openshift-cluster-node-tuning-operator/tuned-vvjl9"
Sep 17 17:41:03.309585 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:03.309474    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mnlzd\" (UniqueName: \"kubernetes.io/projected/5ab7aae3-053b-4e36-9c4a-c523f0cfb615-kube-api-access-mnlzd\") pod \"network-metrics-daemon-xg2wr\" (UID: \"5ab7aae3-053b-4e36-9c4a-c523f0cfb615\") " pod="openshift-multus/network-metrics-daemon-xg2wr"
Sep 17 17:41:03.309875 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:03.309828    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kg27h\" (UniqueName: \"kubernetes.io/projected/0a5f3756-68b8-4e1d-97fc-75a9e6e6fb27-kube-api-access-kg27h\") pod \"ovnkube-node-vwtvz\" (UID: \"0a5f3756-68b8-4e1d-97fc-75a9e6e6fb27\") " pod="openshift-ovn-kubernetes/ovnkube-node-vwtvz"
Sep 17 17:41:03.309875 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:03.309841    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gb66b\" (UniqueName: \"kubernetes.io/projected/7e56ce66-8c6f-4fa0-9560-6210a25593fa-kube-api-access-gb66b\") pod \"aws-ebs-csi-driver-node-6xdld\" (UID: \"7e56ce66-8c6f-4fa0-9560-6210a25593fa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6xdld"
Sep 17 17:41:03.309875 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:03.309843    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xbxgz\" (UniqueName: \"kubernetes.io/projected/e45ad616-0131-4f4e-adf4-b567b76edc38-kube-api-access-xbxgz\") pod \"node-ca-cfn7g\" (UID: \"e45ad616-0131-4f4e-adf4-b567b76edc38\") " pod="openshift-image-registry/node-ca-cfn7g"
Sep 17 17:41:03.311369 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:03.311344    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mmpqq\" (UniqueName: \"kubernetes.io/projected/451c115d-30f1-4679-be41-ff6d2beedaf8-kube-api-access-mmpqq\") pod \"tuned-vvjl9\" (UID: \"451c115d-30f1-4679-be41-ff6d2beedaf8\") " pod="openshift-cluster-node-tuning-operator/tuned-vvjl9"
Sep 17 17:41:03.400629 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:03.400586    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/41a677ee-5541-4579-a6e9-93b915b89d2d-host-var-lib-kubelet\") pod \"multus-dsxjr\" (UID: \"41a677ee-5541-4579-a6e9-93b915b89d2d\") " pod="openshift-multus/multus-dsxjr"
Sep 17 17:41:03.400629 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:03.400626    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j8wst\" (UniqueName: \"kubernetes.io/projected/00f9603c-1639-4399-92ae-c1e4bf354395-kube-api-access-j8wst\") pod \"iptables-alerter-zrbgv\" (UID: \"00f9603c-1639-4399-92ae-c1e4bf354395\") " pod="openshift-network-operator/iptables-alerter-zrbgv"
Sep 17 17:41:03.400870 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:03.400652    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bb03de28-d033-4586-8a80-54465decf01e-system-cni-dir\") pod \"multus-additional-cni-plugins-ptjgb\" (UID: \"bb03de28-d033-4586-8a80-54465decf01e\") " pod="openshift-multus/multus-additional-cni-plugins-ptjgb"
Sep 17 17:41:03.400870 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:03.400694    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bb03de28-d033-4586-8a80-54465decf01e-system-cni-dir\") pod \"multus-additional-cni-plugins-ptjgb\" (UID: \"bb03de28-d033-4586-8a80-54465decf01e\") " pod="openshift-multus/multus-additional-cni-plugins-ptjgb"
Sep 17 17:41:03.400870 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:03.400689    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/41a677ee-5541-4579-a6e9-93b915b89d2d-host-var-lib-kubelet\") pod \"multus-dsxjr\" (UID: \"41a677ee-5541-4579-a6e9-93b915b89d2d\") " pod="openshift-multus/multus-dsxjr"
Sep 17 17:41:03.400870 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:03.400729    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/41a677ee-5541-4579-a6e9-93b915b89d2d-cnibin\") pod \"multus-dsxjr\" (UID: \"41a677ee-5541-4579-a6e9-93b915b89d2d\") " pod="openshift-multus/multus-dsxjr"
Sep 17 17:41:03.400870 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:03.400756    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/41a677ee-5541-4579-a6e9-93b915b89d2d-multus-socket-dir-parent\") pod \"multus-dsxjr\" (UID: \"41a677ee-5541-4579-a6e9-93b915b89d2d\") " pod="openshift-multus/multus-dsxjr"
Sep 17 17:41:03.400870 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:03.400774    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/41a677ee-5541-4579-a6e9-93b915b89d2d-host-var-lib-cni-multus\") pod \"multus-dsxjr\" (UID: \"41a677ee-5541-4579-a6e9-93b915b89d2d\") " pod="openshift-multus/multus-dsxjr"
Sep 17 17:41:03.400870 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:03.400797    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/00f9603c-1639-4399-92ae-c1e4bf354395-host-slash\") pod \"iptables-alerter-zrbgv\" (UID: \"00f9603c-1639-4399-92ae-c1e4bf354395\") " pod="openshift-network-operator/iptables-alerter-zrbgv"
Sep 17 17:41:03.400870 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:03.400803    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/41a677ee-5541-4579-a6e9-93b915b89d2d-cnibin\") pod \"multus-dsxjr\" (UID: \"41a677ee-5541-4579-a6e9-93b915b89d2d\") " pod="openshift-multus/multus-dsxjr"
Sep 17 17:41:03.400870 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:03.400826    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/41a677ee-5541-4579-a6e9-93b915b89d2d-multus-conf-dir\") pod \"multus-dsxjr\" (UID: \"41a677ee-5541-4579-a6e9-93b915b89d2d\") " pod="openshift-multus/multus-dsxjr"
Sep 17 17:41:03.400870 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:03.400845    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/41a677ee-5541-4579-a6e9-93b915b89d2d-multus-socket-dir-parent\") pod \"multus-dsxjr\" (UID: \"41a677ee-5541-4579-a6e9-93b915b89d2d\") " pod="openshift-multus/multus-dsxjr"
Sep 17 17:41:03.400870 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:03.400855    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/41a677ee-5541-4579-a6e9-93b915b89d2d-host-var-lib-cni-multus\") pod \"multus-dsxjr\" (UID: \"41a677ee-5541-4579-a6e9-93b915b89d2d\") " pod="openshift-multus/multus-dsxjr"
Sep 17 17:41:03.400870 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:03.400869    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/41a677ee-5541-4579-a6e9-93b915b89d2d-multus-conf-dir\") pod \"multus-dsxjr\" (UID: \"41a677ee-5541-4579-a6e9-93b915b89d2d\") " pod="openshift-multus/multus-dsxjr"
Sep 17 17:41:03.401354 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:03.400882    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/00f9603c-1639-4399-92ae-c1e4bf354395-host-slash\") pod \"iptables-alerter-zrbgv\" (UID: \"00f9603c-1639-4399-92ae-c1e4bf354395\") " pod="openshift-network-operator/iptables-alerter-zrbgv"
Sep 17 17:41:03.401354 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:03.400930    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/41a677ee-5541-4579-a6e9-93b915b89d2d-multus-daemon-config\") pod \"multus-dsxjr\" (UID: \"41a677ee-5541-4579-a6e9-93b915b89d2d\") " pod="openshift-multus/multus-dsxjr"
Sep 17 17:41:03.401354 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:03.400971    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lndx7\" (UniqueName: \"kubernetes.io/projected/6d0a7973-cc86-4fda-9056-11b5923bd14c-kube-api-access-lndx7\") pod \"network-check-target-4cb4f\" (UID: \"6d0a7973-cc86-4fda-9056-11b5923bd14c\") " pod="openshift-network-diagnostics/network-check-target-4cb4f"
Sep 17 17:41:03.401354 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:03.400997    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bb03de28-d033-4586-8a80-54465decf01e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-ptjgb\" (UID: \"bb03de28-d033-4586-8a80-54465decf01e\") " pod="openshift-multus/multus-additional-cni-plugins-ptjgb"
Sep 17 17:41:03.401354 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:03.401022    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/bb03de28-d033-4586-8a80-54465decf01e-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-ptjgb\" (UID: \"bb03de28-d033-4586-8a80-54465decf01e\") " pod="openshift-multus/multus-additional-cni-plugins-ptjgb"
Sep 17 17:41:03.401354 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:03.401051    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hrmln\" (UniqueName: \"kubernetes.io/projected/19204c4a-e05d-4159-94c8-477036fa461a-kube-api-access-hrmln\") pod \"node-resolver-454w2\" (UID: \"19204c4a-e05d-4159-94c8-477036fa461a\") " pod="openshift-dns/node-resolver-454w2"
Sep 17 17:41:03.401354 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:03.401082    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bb03de28-d033-4586-8a80-54465decf01e-cni-binary-copy\") pod \"multus-additional-cni-plugins-ptjgb\" (UID: \"bb03de28-d033-4586-8a80-54465decf01e\") " pod="openshift-multus/multus-additional-cni-plugins-ptjgb"
Sep 17 17:41:03.401354 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:03.401106    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/19204c4a-e05d-4159-94c8-477036fa461a-hosts-file\") pod \"node-resolver-454w2\" (UID: \"19204c4a-e05d-4159-94c8-477036fa461a\") " pod="openshift-dns/node-resolver-454w2"
Sep 17 17:41:03.401354 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:03.401129    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/41a677ee-5541-4579-a6e9-93b915b89d2d-multus-cni-dir\") pod \"multus-dsxjr\" (UID: \"41a677ee-5541-4579-a6e9-93b915b89d2d\") " pod="openshift-multus/multus-dsxjr"
Sep 17 17:41:03.401354 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:03.401155    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mn68f\" (UniqueName: \"kubernetes.io/projected/bb03de28-d033-4586-8a80-54465decf01e-kube-api-access-mn68f\") pod \"multus-additional-cni-plugins-ptjgb\" (UID: \"bb03de28-d033-4586-8a80-54465decf01e\") " pod="openshift-multus/multus-additional-cni-plugins-ptjgb"
Sep 17 17:41:03.401354 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:03.401179    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/41a677ee-5541-4579-a6e9-93b915b89d2d-host-var-lib-cni-bin\") pod \"multus-dsxjr\" (UID: \"41a677ee-5541-4579-a6e9-93b915b89d2d\") " pod="openshift-multus/multus-dsxjr"
Sep 17 17:41:03.401354 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:03.401201    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bb03de28-d033-4586-8a80-54465decf01e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-ptjgb\" (UID: \"bb03de28-d033-4586-8a80-54465decf01e\") " pod="openshift-multus/multus-additional-cni-plugins-ptjgb"
Sep 17 17:41:03.401354 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:03.401203    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nw6pc\" (UniqueName: \"kubernetes.io/projected/41a677ee-5541-4579-a6e9-93b915b89d2d-kube-api-access-nw6pc\") pod \"multus-dsxjr\" (UID: \"41a677ee-5541-4579-a6e9-93b915b89d2d\") " pod="openshift-multus/multus-dsxjr"
Sep 17 17:41:03.401354 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:03.401251    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/41a677ee-5541-4579-a6e9-93b915b89d2d-system-cni-dir\") pod \"multus-dsxjr\" (UID: \"41a677ee-5541-4579-a6e9-93b915b89d2d\") " pod="openshift-multus/multus-dsxjr"
Sep 17 17:41:03.401354 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:03.401278    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/41a677ee-5541-4579-a6e9-93b915b89d2d-host-run-multus-certs\") pod \"multus-dsxjr\" (UID: \"41a677ee-5541-4579-a6e9-93b915b89d2d\") " pod="openshift-multus/multus-dsxjr"
Sep 17 17:41:03.401354 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:03.401304    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/41a677ee-5541-4579-a6e9-93b915b89d2d-etc-kubernetes\") pod \"multus-dsxjr\" (UID: \"41a677ee-5541-4579-a6e9-93b915b89d2d\") " pod="openshift-multus/multus-dsxjr"
Sep 17 17:41:03.401354 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:03.401314    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/19204c4a-e05d-4159-94c8-477036fa461a-hosts-file\") pod \"node-resolver-454w2\" (UID: \"19204c4a-e05d-4159-94c8-477036fa461a\") " pod="openshift-dns/node-resolver-454w2"
Sep 17 17:41:03.402231 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:03.401329    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bb03de28-d033-4586-8a80-54465decf01e-cnibin\") pod \"multus-additional-cni-plugins-ptjgb\" (UID: \"bb03de28-d033-4586-8a80-54465decf01e\") " pod="openshift-multus/multus-additional-cni-plugins-ptjgb"
Sep 17 17:41:03.402231 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:03.401370    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bb03de28-d033-4586-8a80-54465decf01e-cnibin\") pod \"multus-additional-cni-plugins-ptjgb\" (UID: \"bb03de28-d033-4586-8a80-54465decf01e\") " pod="openshift-multus/multus-additional-cni-plugins-ptjgb"
Sep 17 17:41:03.402231 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:03.401417    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/41a677ee-5541-4579-a6e9-93b915b89d2d-system-cni-dir\") pod \"multus-dsxjr\" (UID: \"41a677ee-5541-4579-a6e9-93b915b89d2d\") " pod="openshift-multus/multus-dsxjr"
Sep 17 17:41:03.402231 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:03.401450    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/41a677ee-5541-4579-a6e9-93b915b89d2d-host-var-lib-cni-bin\") pod \"multus-dsxjr\" (UID: \"41a677ee-5541-4579-a6e9-93b915b89d2d\") " pod="openshift-multus/multus-dsxjr"
Sep 17 17:41:03.402231 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:03.401454    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/41a677ee-5541-4579-a6e9-93b915b89d2d-host-run-multus-certs\") pod \"multus-dsxjr\" (UID: \"41a677ee-5541-4579-a6e9-93b915b89d2d\") " pod="openshift-multus/multus-dsxjr"
Sep 17 17:41:03.402231 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:03.401482    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/41a677ee-5541-4579-a6e9-93b915b89d2d-etc-kubernetes\") pod \"multus-dsxjr\" (UID: \"41a677ee-5541-4579-a6e9-93b915b89d2d\") " pod="openshift-multus/multus-dsxjr"
Sep 17 17:41:03.402231 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:03.401488    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/41a677ee-5541-4579-a6e9-93b915b89d2d-multus-daemon-config\") pod \"multus-dsxjr\" (UID: \"41a677ee-5541-4579-a6e9-93b915b89d2d\") " pod="openshift-multus/multus-dsxjr"
Sep 17 17:41:03.402231 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:03.401510    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/41a677ee-5541-4579-a6e9-93b915b89d2d-host-run-netns\") pod \"multus-dsxjr\" (UID: \"41a677ee-5541-4579-a6e9-93b915b89d2d\") " pod="openshift-multus/multus-dsxjr"
Sep 17 17:41:03.402231 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:03.401549    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/41a677ee-5541-4579-a6e9-93b915b89d2d-multus-cni-dir\") pod \"multus-dsxjr\" (UID: \"41a677ee-5541-4579-a6e9-93b915b89d2d\") " pod="openshift-multus/multus-dsxjr"
Sep 17 17:41:03.402231 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:03.401556    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/bb03de28-d033-4586-8a80-54465decf01e-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-ptjgb\" (UID: \"bb03de28-d033-4586-8a80-54465decf01e\") " pod="openshift-multus/multus-additional-cni-plugins-ptjgb"
Sep 17 17:41:03.402231 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:03.401565    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/bb03de28-d033-4586-8a80-54465decf01e-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-ptjgb\" (UID: \"bb03de28-d033-4586-8a80-54465decf01e\") " pod="openshift-multus/multus-additional-cni-plugins-ptjgb"
Sep 17 17:41:03.402231 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:03.401587    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/00f9603c-1639-4399-92ae-c1e4bf354395-iptables-alerter-script\") pod \"iptables-alerter-zrbgv\" (UID: \"00f9603c-1639-4399-92ae-c1e4bf354395\") " pod="openshift-network-operator/iptables-alerter-zrbgv"
Sep 17 17:41:03.402231 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:03.401646    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/41a677ee-5541-4579-a6e9-93b915b89d2d-hostroot\") pod \"multus-dsxjr\" (UID: \"41a677ee-5541-4579-a6e9-93b915b89d2d\") " pod="openshift-multus/multus-dsxjr"
Sep 17 17:41:03.402231 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:03.401588    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/41a677ee-5541-4579-a6e9-93b915b89d2d-host-run-netns\") pod \"multus-dsxjr\" (UID: \"41a677ee-5541-4579-a6e9-93b915b89d2d\") " pod="openshift-multus/multus-dsxjr"
Sep 17 17:41:03.402231 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:03.401698    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/41a677ee-5541-4579-a6e9-93b915b89d2d-hostroot\") pod \"multus-dsxjr\" (UID: \"41a677ee-5541-4579-a6e9-93b915b89d2d\") " pod="openshift-multus/multus-dsxjr"
Sep 17 17:41:03.402231 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:03.401706    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/9a0f326b-5e07-4829-bc1f-084edb7f3b17-konnectivity-ca\") pod \"konnectivity-agent-j5jw8\" (UID: \"9a0f326b-5e07-4829-bc1f-084edb7f3b17\") " pod="kube-system/konnectivity-agent-j5jw8"
Sep 17 17:41:03.402231 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:03.401743    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/9a0f326b-5e07-4829-bc1f-084edb7f3b17-agent-certs\") pod \"konnectivity-agent-j5jw8\" (UID: \"9a0f326b-5e07-4829-bc1f-084edb7f3b17\") " pod="kube-system/konnectivity-agent-j5jw8"
Sep 17 17:41:03.402231 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:03.401771    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/19204c4a-e05d-4159-94c8-477036fa461a-tmp-dir\") pod \"node-resolver-454w2\" (UID: \"19204c4a-e05d-4159-94c8-477036fa461a\") " pod="openshift-dns/node-resolver-454w2"
Sep 17 17:41:03.402969 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:03.402027    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bb03de28-d033-4586-8a80-54465decf01e-cni-binary-copy\") pod \"multus-additional-cni-plugins-ptjgb\" (UID: \"bb03de28-d033-4586-8a80-54465decf01e\") " pod="openshift-multus/multus-additional-cni-plugins-ptjgb"
Sep 17 17:41:03.402969 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:03.402052    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/00f9603c-1639-4399-92ae-c1e4bf354395-iptables-alerter-script\") pod \"iptables-alerter-zrbgv\" (UID: \"00f9603c-1639-4399-92ae-c1e4bf354395\") " pod="openshift-network-operator/iptables-alerter-zrbgv"
Sep 17 17:41:03.402969 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:03.402092    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/19204c4a-e05d-4159-94c8-477036fa461a-tmp-dir\") pod \"node-resolver-454w2\" (UID: \"19204c4a-e05d-4159-94c8-477036fa461a\") " pod="openshift-dns/node-resolver-454w2"
Sep 17 17:41:03.402969 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:03.402106    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/41a677ee-5541-4579-a6e9-93b915b89d2d-cni-binary-copy\") pod \"multus-dsxjr\" (UID: \"41a677ee-5541-4579-a6e9-93b915b89d2d\") " pod="openshift-multus/multus-dsxjr"
Sep 17 17:41:03.402969 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:03.402110    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/bb03de28-d033-4586-8a80-54465decf01e-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-ptjgb\" (UID: \"bb03de28-d033-4586-8a80-54465decf01e\") " pod="openshift-multus/multus-additional-cni-plugins-ptjgb"
Sep 17 17:41:03.402969 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:03.402137    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/41a677ee-5541-4579-a6e9-93b915b89d2d-host-run-k8s-cni-cncf-io\") pod \"multus-dsxjr\" (UID: \"41a677ee-5541-4579-a6e9-93b915b89d2d\") " pod="openshift-multus/multus-dsxjr"
Sep 17 17:41:03.402969 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:03.402195    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/41a677ee-5541-4579-a6e9-93b915b89d2d-host-run-k8s-cni-cncf-io\") pod \"multus-dsxjr\" (UID: \"41a677ee-5541-4579-a6e9-93b915b89d2d\") " pod="openshift-multus/multus-dsxjr"
Sep 17 17:41:03.402969 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:03.402508    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/9a0f326b-5e07-4829-bc1f-084edb7f3b17-konnectivity-ca\") pod \"konnectivity-agent-j5jw8\" (UID: \"9a0f326b-5e07-4829-bc1f-084edb7f3b17\") " pod="kube-system/konnectivity-agent-j5jw8"
Sep 17 17:41:03.402969 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:03.402554    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/41a677ee-5541-4579-a6e9-93b915b89d2d-cni-binary-copy\") pod \"multus-dsxjr\" (UID: \"41a677ee-5541-4579-a6e9-93b915b89d2d\") " pod="openshift-multus/multus-dsxjr"
Sep 17 17:41:03.404618 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:03.404597    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/9a0f326b-5e07-4829-bc1f-084edb7f3b17-agent-certs\") pod \"konnectivity-agent-j5jw8\" (UID: \"9a0f326b-5e07-4829-bc1f-084edb7f3b17\") " pod="kube-system/konnectivity-agent-j5jw8"
Sep 17 17:41:03.409769 ip-10-0-139-237 kubenswrapper[2567]: E0917 17:41:03.409747    2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 17:41:03.409769 ip-10-0-139-237 kubenswrapper[2567]: E0917 17:41:03.409772    2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 17:41:03.409944 ip-10-0-139-237 kubenswrapper[2567]: E0917 17:41:03.409794    2567 projected.go:196] Error preparing data for projected volume kube-api-access-lndx7 for pod openshift-network-diagnostics/network-check-target-4cb4f: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:41:03.409944 ip-10-0-139-237 kubenswrapper[2567]: E0917 17:41:03.409914    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6d0a7973-cc86-4fda-9056-11b5923bd14c-kube-api-access-lndx7 podName:6d0a7973-cc86-4fda-9056-11b5923bd14c nodeName:}" failed. No retries permitted until 2026-09-17 17:41:03.909895309 +0000 UTC m=+4.933500450 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-lndx7" (UniqueName: "kubernetes.io/projected/6d0a7973-cc86-4fda-9056-11b5923bd14c-kube-api-access-lndx7") pod "network-check-target-4cb4f" (UID: "6d0a7973-cc86-4fda-9056-11b5923bd14c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:41:03.411853 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:03.411829    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hrmln\" (UniqueName: \"kubernetes.io/projected/19204c4a-e05d-4159-94c8-477036fa461a-kube-api-access-hrmln\") pod \"node-resolver-454w2\" (UID: \"19204c4a-e05d-4159-94c8-477036fa461a\") " pod="openshift-dns/node-resolver-454w2"
Sep 17 17:41:03.412179 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:03.412138    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nw6pc\" (UniqueName: \"kubernetes.io/projected/41a677ee-5541-4579-a6e9-93b915b89d2d-kube-api-access-nw6pc\") pod \"multus-dsxjr\" (UID: \"41a677ee-5541-4579-a6e9-93b915b89d2d\") " pod="openshift-multus/multus-dsxjr"
Sep 17 17:41:03.412415 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:03.412397    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j8wst\" (UniqueName: \"kubernetes.io/projected/00f9603c-1639-4399-92ae-c1e4bf354395-kube-api-access-j8wst\") pod \"iptables-alerter-zrbgv\" (UID: \"00f9603c-1639-4399-92ae-c1e4bf354395\") " pod="openshift-network-operator/iptables-alerter-zrbgv"
Sep 17 17:41:03.413261 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:03.413242    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mn68f\" (UniqueName: \"kubernetes.io/projected/bb03de28-d033-4586-8a80-54465decf01e-kube-api-access-mn68f\") pod \"multus-additional-cni-plugins-ptjgb\" (UID: \"bb03de28-d033-4586-8a80-54465decf01e\") " pod="openshift-multus/multus-additional-cni-plugins-ptjgb"
Sep 17 17:41:03.490479 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:03.490408    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-cfn7g"
Sep 17 17:41:03.502924 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:03.502897    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6xdld"
Sep 17 17:41:03.512864 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:03.512842    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-vwtvz"
Sep 17 17:41:03.519647 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:03.519628    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-vvjl9"
Sep 17 17:41:03.527456 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:03.527433    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-454w2"
Sep 17 17:41:03.535029 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:03.535011    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-dsxjr"
Sep 17 17:41:03.542953 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:03.542936    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-zrbgv"
Sep 17 17:41:03.543230 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:03.543213    2567 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 17:41:03.550642 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:03.550623    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-j5jw8"
Sep 17 17:41:03.557130 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:03.557112    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-ptjgb"
Sep 17 17:41:03.804850 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:03.804773    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/db73a8fe-d34c-41c8-8ba4-8040b4bacf13-original-pull-secret\") pod \"global-pull-secret-syncer-vjstm\" (UID: \"db73a8fe-d34c-41c8-8ba4-8040b4bacf13\") " pod="kube-system/global-pull-secret-syncer-vjstm"
Sep 17 17:41:03.804850 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:03.804823    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5ab7aae3-053b-4e36-9c4a-c523f0cfb615-metrics-certs\") pod \"network-metrics-daemon-xg2wr\" (UID: \"5ab7aae3-053b-4e36-9c4a-c523f0cfb615\") " pod="openshift-multus/network-metrics-daemon-xg2wr"
Sep 17 17:41:03.805069 ip-10-0-139-237 kubenswrapper[2567]: E0917 17:41:03.804940    2567 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 17:41:03.805069 ip-10-0-139-237 kubenswrapper[2567]: E0917 17:41:03.804965    2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:41:03.805069 ip-10-0-139-237 kubenswrapper[2567]: E0917 17:41:03.805015    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/db73a8fe-d34c-41c8-8ba4-8040b4bacf13-original-pull-secret podName:db73a8fe-d34c-41c8-8ba4-8040b4bacf13 nodeName:}" failed. No retries permitted until 2026-09-17 17:41:04.804996035 +0000 UTC m=+5.828601153 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/db73a8fe-d34c-41c8-8ba4-8040b4bacf13-original-pull-secret") pod "global-pull-secret-syncer-vjstm" (UID: "db73a8fe-d34c-41c8-8ba4-8040b4bacf13") : object "kube-system"/"original-pull-secret" not registered
Sep 17 17:41:03.805069 ip-10-0-139-237 kubenswrapper[2567]: E0917 17:41:03.805035    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5ab7aae3-053b-4e36-9c4a-c523f0cfb615-metrics-certs podName:5ab7aae3-053b-4e36-9c4a-c523f0cfb615 nodeName:}" failed. No retries permitted until 2026-09-17 17:41:04.805026125 +0000 UTC m=+5.828631239 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5ab7aae3-053b-4e36-9c4a-c523f0cfb615-metrics-certs") pod "network-metrics-daemon-xg2wr" (UID: "5ab7aae3-053b-4e36-9c4a-c523f0cfb615") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:41:04.006878 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:04.006845    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lndx7\" (UniqueName: \"kubernetes.io/projected/6d0a7973-cc86-4fda-9056-11b5923bd14c-kube-api-access-lndx7\") pod \"network-check-target-4cb4f\" (UID: \"6d0a7973-cc86-4fda-9056-11b5923bd14c\") " pod="openshift-network-diagnostics/network-check-target-4cb4f"
Sep 17 17:41:04.007037 ip-10-0-139-237 kubenswrapper[2567]: E0917 17:41:04.006997    2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 17:41:04.007037 ip-10-0-139-237 kubenswrapper[2567]: E0917 17:41:04.007016    2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 17:41:04.007037 ip-10-0-139-237 kubenswrapper[2567]: E0917 17:41:04.007025    2567 projected.go:196] Error preparing data for projected volume kube-api-access-lndx7 for pod openshift-network-diagnostics/network-check-target-4cb4f: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:41:04.007142 ip-10-0-139-237 kubenswrapper[2567]: E0917 17:41:04.007080    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6d0a7973-cc86-4fda-9056-11b5923bd14c-kube-api-access-lndx7 podName:6d0a7973-cc86-4fda-9056-11b5923bd14c nodeName:}" failed. No retries permitted until 2026-09-17 17:41:05.007062889 +0000 UTC m=+6.030668020 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-lndx7" (UniqueName: "kubernetes.io/projected/6d0a7973-cc86-4fda-9056-11b5923bd14c-kube-api-access-lndx7") pod "network-check-target-4cb4f" (UID: "6d0a7973-cc86-4fda-9056-11b5923bd14c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:41:04.244054 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:04.243968    2567 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-16 17:36:03 +0000 UTC" deadline="2028-04-13 01:48:51.152794337 +0000 UTC"
Sep 17 17:41:04.244054 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:04.244005    2567 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13760h7m46.908794119s"
Sep 17 17:41:04.281500 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:04.281465    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vjstm"
Sep 17 17:41:04.281646 ip-10-0-139-237 kubenswrapper[2567]: E0917 17:41:04.281588    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vjstm" podUID="db73a8fe-d34c-41c8-8ba4-8040b4bacf13"
Sep 17 17:41:04.327018 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:04.326640    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a5f3756_68b8_4e1d_97fc_75a9e6e6fb27.slice/crio-6833cff9bc0a3289db25b59c0c25a1822d42ba7d0e3d263bfdf08d303a4c1d3f WatchSource:0}: Error finding container 6833cff9bc0a3289db25b59c0c25a1822d42ba7d0e3d263bfdf08d303a4c1d3f: Status 404 returned error can't find the container with id 6833cff9bc0a3289db25b59c0c25a1822d42ba7d0e3d263bfdf08d303a4c1d3f
Sep 17 17:41:04.328322 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:04.328291    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a0f326b_5e07_4829_bc1f_084edb7f3b17.slice/crio-4d746de4517bb04b61bbcafe1a3eb7cd8f3cad2b966befa1eefeb85edc498fd8 WatchSource:0}: Error finding container 4d746de4517bb04b61bbcafe1a3eb7cd8f3cad2b966befa1eefeb85edc498fd8: Status 404 returned error can't find the container with id 4d746de4517bb04b61bbcafe1a3eb7cd8f3cad2b966befa1eefeb85edc498fd8
Sep 17 17:41:04.329099 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:04.329071    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19204c4a_e05d_4159_94c8_477036fa461a.slice/crio-036996d0980f92ebbe2bbb338782fd0ccf141f57792be4ee49b5de0c27d0e9c4 WatchSource:0}: Error finding container 036996d0980f92ebbe2bbb338782fd0ccf141f57792be4ee49b5de0c27d0e9c4: Status 404 returned error can't find the container with id 036996d0980f92ebbe2bbb338782fd0ccf141f57792be4ee49b5de0c27d0e9c4
Sep 17 17:41:04.330433 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:04.330377    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb03de28_d033_4586_8a80_54465decf01e.slice/crio-2534be6d7cddb0bb2bdec9dead725b04004fc7e41318429b016641b855a8d284 WatchSource:0}: Error finding container 2534be6d7cddb0bb2bdec9dead725b04004fc7e41318429b016641b855a8d284: Status 404 returned error can't find the container with id 2534be6d7cddb0bb2bdec9dead725b04004fc7e41318429b016641b855a8d284
Sep 17 17:41:04.332875 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:04.332830    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e56ce66_8c6f_4fa0_9560_6210a25593fa.slice/crio-ede548e6cce8b064c4f2757549092a7fa1c2886aa5404ad5a9d4acd61060d16c WatchSource:0}: Error finding container ede548e6cce8b064c4f2757549092a7fa1c2886aa5404ad5a9d4acd61060d16c: Status 404 returned error can't find the container with id ede548e6cce8b064c4f2757549092a7fa1c2886aa5404ad5a9d4acd61060d16c
Sep 17 17:41:04.333702 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:04.333672    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode45ad616_0131_4f4e_adf4_b567b76edc38.slice/crio-83701125dfa76fc840686552a8b73a46eb9e3632c2d4f1feefeb1af2126b7203 WatchSource:0}: Error finding container 83701125dfa76fc840686552a8b73a46eb9e3632c2d4f1feefeb1af2126b7203: Status 404 returned error can't find the container with id 83701125dfa76fc840686552a8b73a46eb9e3632c2d4f1feefeb1af2126b7203
Sep 17 17:41:04.334416 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:04.334312    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod451c115d_30f1_4679_be41_ff6d2beedaf8.slice/crio-90c022430137b74855e854f064e6e2dd7cf7f9ac74de3d54c73c8b1f6a7e7525 WatchSource:0}: Error finding container 90c022430137b74855e854f064e6e2dd7cf7f9ac74de3d54c73c8b1f6a7e7525: Status 404 returned error can't find the container with id 90c022430137b74855e854f064e6e2dd7cf7f9ac74de3d54c73c8b1f6a7e7525
Sep 17 17:41:04.338061 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:04.335763    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41a677ee_5541_4579_a6e9_93b915b89d2d.slice/crio-661f8251cbccfb0824a08b318f389a31f785da8a19743e3edfe273c03e09c2a0 WatchSource:0}: Error finding container 661f8251cbccfb0824a08b318f389a31f785da8a19743e3edfe273c03e09c2a0: Status 404 returned error can't find the container with id 661f8251cbccfb0824a08b318f389a31f785da8a19743e3edfe273c03e09c2a0
Sep 17 17:41:04.338061 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:04.336884    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00f9603c_1639_4399_92ae_c1e4bf354395.slice/crio-8075516aedc69baba37a45c2b2fdab25d2f4e6d6f879ac781076cd4ab5f8ef5f WatchSource:0}: Error finding container 8075516aedc69baba37a45c2b2fdab25d2f4e6d6f879ac781076cd4ab5f8ef5f: Status 404 returned error can't find the container with id 8075516aedc69baba37a45c2b2fdab25d2f4e6d6f879ac781076cd4ab5f8ef5f
Sep 17 17:41:04.813291 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:04.812480    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/db73a8fe-d34c-41c8-8ba4-8040b4bacf13-original-pull-secret\") pod \"global-pull-secret-syncer-vjstm\" (UID: \"db73a8fe-d34c-41c8-8ba4-8040b4bacf13\") " pod="kube-system/global-pull-secret-syncer-vjstm"
Sep 17 17:41:04.813291 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:04.812541    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5ab7aae3-053b-4e36-9c4a-c523f0cfb615-metrics-certs\") pod \"network-metrics-daemon-xg2wr\" (UID: \"5ab7aae3-053b-4e36-9c4a-c523f0cfb615\") " pod="openshift-multus/network-metrics-daemon-xg2wr"
Sep 17 17:41:04.813291 ip-10-0-139-237 kubenswrapper[2567]: E0917 17:41:04.812738    2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:41:04.813291 ip-10-0-139-237 kubenswrapper[2567]: E0917 17:41:04.812798    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5ab7aae3-053b-4e36-9c4a-c523f0cfb615-metrics-certs podName:5ab7aae3-053b-4e36-9c4a-c523f0cfb615 nodeName:}" failed. No retries permitted until 2026-09-17 17:41:06.812781165 +0000 UTC m=+7.836386284 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5ab7aae3-053b-4e36-9c4a-c523f0cfb615-metrics-certs") pod "network-metrics-daemon-xg2wr" (UID: "5ab7aae3-053b-4e36-9c4a-c523f0cfb615") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:41:04.813291 ip-10-0-139-237 kubenswrapper[2567]: E0917 17:41:04.813167    2567 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 17:41:04.813291 ip-10-0-139-237 kubenswrapper[2567]: E0917 17:41:04.813215    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/db73a8fe-d34c-41c8-8ba4-8040b4bacf13-original-pull-secret podName:db73a8fe-d34c-41c8-8ba4-8040b4bacf13 nodeName:}" failed. No retries permitted until 2026-09-17 17:41:06.813199892 +0000 UTC m=+7.836805010 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/db73a8fe-d34c-41c8-8ba4-8040b4bacf13-original-pull-secret") pod "global-pull-secret-syncer-vjstm" (UID: "db73a8fe-d34c-41c8-8ba4-8040b4bacf13") : object "kube-system"/"original-pull-secret" not registered
Sep 17 17:41:05.014377 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:05.014332    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lndx7\" (UniqueName: \"kubernetes.io/projected/6d0a7973-cc86-4fda-9056-11b5923bd14c-kube-api-access-lndx7\") pod \"network-check-target-4cb4f\" (UID: \"6d0a7973-cc86-4fda-9056-11b5923bd14c\") " pod="openshift-network-diagnostics/network-check-target-4cb4f"
Sep 17 17:41:05.014572 ip-10-0-139-237 kubenswrapper[2567]: E0917 17:41:05.014518    2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 17:41:05.014572 ip-10-0-139-237 kubenswrapper[2567]: E0917 17:41:05.014551    2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 17:41:05.014572 ip-10-0-139-237 kubenswrapper[2567]: E0917 17:41:05.014564    2567 projected.go:196] Error preparing data for projected volume kube-api-access-lndx7 for pod openshift-network-diagnostics/network-check-target-4cb4f: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:41:05.014743 ip-10-0-139-237 kubenswrapper[2567]: E0917 17:41:05.014621    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6d0a7973-cc86-4fda-9056-11b5923bd14c-kube-api-access-lndx7 podName:6d0a7973-cc86-4fda-9056-11b5923bd14c nodeName:}" failed. No retries permitted until 2026-09-17 17:41:07.01460361 +0000 UTC m=+8.038208746 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-lndx7" (UniqueName: "kubernetes.io/projected/6d0a7973-cc86-4fda-9056-11b5923bd14c-kube-api-access-lndx7") pod "network-check-target-4cb4f" (UID: "6d0a7973-cc86-4fda-9056-11b5923bd14c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:41:05.244443 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:05.244145    2567 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-16 17:36:03 +0000 UTC" deadline="2028-05-07 01:53:30.709921496 +0000 UTC"
Sep 17 17:41:05.244443 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:05.244395    2567 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14336h12m25.465532609s"
Sep 17 17:41:05.281986 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:05.281931    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4cb4f"
Sep 17 17:41:05.282240 ip-10-0-139-237 kubenswrapper[2567]: E0917 17:41:05.282060    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4cb4f" podUID="6d0a7973-cc86-4fda-9056-11b5923bd14c"
Sep 17 17:41:05.282492 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:05.282469    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xg2wr"
Sep 17 17:41:05.282628 ip-10-0-139-237 kubenswrapper[2567]: E0917 17:41:05.282607    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xg2wr" podUID="5ab7aae3-053b-4e36-9c4a-c523f0cfb615"
Sep 17 17:41:05.303922 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:05.303888    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6xdld" event={"ID":"7e56ce66-8c6f-4fa0-9560-6210a25593fa","Type":"ContainerStarted","Data":"ede548e6cce8b064c4f2757549092a7fa1c2886aa5404ad5a9d4acd61060d16c"}
Sep 17 17:41:05.308729 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:05.308697    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ptjgb" event={"ID":"bb03de28-d033-4586-8a80-54465decf01e","Type":"ContainerStarted","Data":"2534be6d7cddb0bb2bdec9dead725b04004fc7e41318429b016641b855a8d284"}
Sep 17 17:41:05.318470 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:05.318440    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vwtvz" event={"ID":"0a5f3756-68b8-4e1d-97fc-75a9e6e6fb27","Type":"ContainerStarted","Data":"6833cff9bc0a3289db25b59c0c25a1822d42ba7d0e3d263bfdf08d303a4c1d3f"}
Sep 17 17:41:05.330085 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:05.329424    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-237.ec2.internal" event={"ID":"840f4093b21dc0fac911784fe7877e0e","Type":"ContainerStarted","Data":"e95d770fec75310f8e8d9aae91572f15e3b9f71b474b544e15bbf4d5df0d68fa"}
Sep 17 17:41:05.346819 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:05.346769    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-237.ec2.internal" podStartSLOduration=3.346750628 podStartE2EDuration="3.346750628s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:41:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:41:05.346398999 +0000 UTC m=+6.370004136" watchObservedRunningTime="2026-09-17 17:41:05.346750628 +0000 UTC m=+6.370355766"
Sep 17 17:41:05.351698 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:05.351556    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-zrbgv" event={"ID":"00f9603c-1639-4399-92ae-c1e4bf354395","Type":"ContainerStarted","Data":"8075516aedc69baba37a45c2b2fdab25d2f4e6d6f879ac781076cd4ab5f8ef5f"}
Sep 17 17:41:05.361117 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:05.361084    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-dsxjr" event={"ID":"41a677ee-5541-4579-a6e9-93b915b89d2d","Type":"ContainerStarted","Data":"661f8251cbccfb0824a08b318f389a31f785da8a19743e3edfe273c03e09c2a0"}
Sep 17 17:41:05.368051 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:05.368020    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-vvjl9" event={"ID":"451c115d-30f1-4679-be41-ff6d2beedaf8","Type":"ContainerStarted","Data":"90c022430137b74855e854f064e6e2dd7cf7f9ac74de3d54c73c8b1f6a7e7525"}
Sep 17 17:41:05.371292 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:05.371262    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-454w2" event={"ID":"19204c4a-e05d-4159-94c8-477036fa461a","Type":"ContainerStarted","Data":"036996d0980f92ebbe2bbb338782fd0ccf141f57792be4ee49b5de0c27d0e9c4"}
Sep 17 17:41:05.377459 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:05.377433    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-j5jw8" event={"ID":"9a0f326b-5e07-4829-bc1f-084edb7f3b17","Type":"ContainerStarted","Data":"4d746de4517bb04b61bbcafe1a3eb7cd8f3cad2b966befa1eefeb85edc498fd8"}
Sep 17 17:41:05.399092 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:05.399062    2567 generic.go:356] "Generic (PLEG): container finished" podID="e678eb327b2639be46e3883082cf83ef" containerID="f957e2e2daaae78c60c07e1d505224381839d5392091f165f10d596856fa0781" exitCode=0
Sep 17 17:41:05.399215 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:05.399166    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-237.ec2.internal" event={"ID":"e678eb327b2639be46e3883082cf83ef","Type":"ContainerDied","Data":"f957e2e2daaae78c60c07e1d505224381839d5392091f165f10d596856fa0781"}
Sep 17 17:41:05.418012 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:05.417985    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-cfn7g" event={"ID":"e45ad616-0131-4f4e-adf4-b567b76edc38","Type":"ContainerStarted","Data":"83701125dfa76fc840686552a8b73a46eb9e3632c2d4f1feefeb1af2126b7203"}
Sep 17 17:41:06.280893 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:06.280836    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vjstm"
Sep 17 17:41:06.281323 ip-10-0-139-237 kubenswrapper[2567]: E0917 17:41:06.281060    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vjstm" podUID="db73a8fe-d34c-41c8-8ba4-8040b4bacf13"
Sep 17 17:41:06.429707 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:06.429631    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-237.ec2.internal" event={"ID":"e678eb327b2639be46e3883082cf83ef","Type":"ContainerStarted","Data":"1b6fdf04c4b93a2d8c0a4bccaab0a7da9f460b6c820a48fe8eea2dbed26c8bec"}
Sep 17 17:41:06.450598 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:06.447921    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-237.ec2.internal" podStartSLOduration=4.447899556 podStartE2EDuration="4.447899556s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:41:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:41:06.446926476 +0000 UTC m=+7.470531640" watchObservedRunningTime="2026-09-17 17:41:06.447899556 +0000 UTC m=+7.471504692"
Sep 17 17:41:06.829380 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:06.829106    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/db73a8fe-d34c-41c8-8ba4-8040b4bacf13-original-pull-secret\") pod \"global-pull-secret-syncer-vjstm\" (UID: \"db73a8fe-d34c-41c8-8ba4-8040b4bacf13\") " pod="kube-system/global-pull-secret-syncer-vjstm"
Sep 17 17:41:06.829380 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:06.829156    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5ab7aae3-053b-4e36-9c4a-c523f0cfb615-metrics-certs\") pod \"network-metrics-daemon-xg2wr\" (UID: \"5ab7aae3-053b-4e36-9c4a-c523f0cfb615\") " pod="openshift-multus/network-metrics-daemon-xg2wr"
Sep 17 17:41:06.829380 ip-10-0-139-237 kubenswrapper[2567]: E0917 17:41:06.829273    2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:41:06.829380 ip-10-0-139-237 kubenswrapper[2567]: E0917 17:41:06.829333    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5ab7aae3-053b-4e36-9c4a-c523f0cfb615-metrics-certs podName:5ab7aae3-053b-4e36-9c4a-c523f0cfb615 nodeName:}" failed. No retries permitted until 2026-09-17 17:41:10.829315482 +0000 UTC m=+11.852920619 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5ab7aae3-053b-4e36-9c4a-c523f0cfb615-metrics-certs") pod "network-metrics-daemon-xg2wr" (UID: "5ab7aae3-053b-4e36-9c4a-c523f0cfb615") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:41:06.829380 ip-10-0-139-237 kubenswrapper[2567]: E0917 17:41:06.829393    2567 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 17:41:06.829839 ip-10-0-139-237 kubenswrapper[2567]: E0917 17:41:06.829423    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/db73a8fe-d34c-41c8-8ba4-8040b4bacf13-original-pull-secret podName:db73a8fe-d34c-41c8-8ba4-8040b4bacf13 nodeName:}" failed. No retries permitted until 2026-09-17 17:41:10.829413394 +0000 UTC m=+11.853018508 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/db73a8fe-d34c-41c8-8ba4-8040b4bacf13-original-pull-secret") pod "global-pull-secret-syncer-vjstm" (UID: "db73a8fe-d34c-41c8-8ba4-8040b4bacf13") : object "kube-system"/"original-pull-secret" not registered
Sep 17 17:41:07.030963 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:07.030928    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lndx7\" (UniqueName: \"kubernetes.io/projected/6d0a7973-cc86-4fda-9056-11b5923bd14c-kube-api-access-lndx7\") pod \"network-check-target-4cb4f\" (UID: \"6d0a7973-cc86-4fda-9056-11b5923bd14c\") " pod="openshift-network-diagnostics/network-check-target-4cb4f"
Sep 17 17:41:07.031150 ip-10-0-139-237 kubenswrapper[2567]: E0917 17:41:07.031134    2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 17:41:07.031208 ip-10-0-139-237 kubenswrapper[2567]: E0917 17:41:07.031156    2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 17:41:07.031208 ip-10-0-139-237 kubenswrapper[2567]: E0917 17:41:07.031171    2567 projected.go:196] Error preparing data for projected volume kube-api-access-lndx7 for pod openshift-network-diagnostics/network-check-target-4cb4f: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:41:07.031316 ip-10-0-139-237 kubenswrapper[2567]: E0917 17:41:07.031227    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6d0a7973-cc86-4fda-9056-11b5923bd14c-kube-api-access-lndx7 podName:6d0a7973-cc86-4fda-9056-11b5923bd14c nodeName:}" failed. No retries permitted until 2026-09-17 17:41:11.031209025 +0000 UTC m=+12.054814143 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-lndx7" (UniqueName: "kubernetes.io/projected/6d0a7973-cc86-4fda-9056-11b5923bd14c-kube-api-access-lndx7") pod "network-check-target-4cb4f" (UID: "6d0a7973-cc86-4fda-9056-11b5923bd14c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:41:07.281624 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:07.281373    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xg2wr"
Sep 17 17:41:07.281624 ip-10-0-139-237 kubenswrapper[2567]: E0917 17:41:07.281524    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xg2wr" podUID="5ab7aae3-053b-4e36-9c4a-c523f0cfb615"
Sep 17 17:41:07.282179 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:07.281994    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4cb4f"
Sep 17 17:41:07.282179 ip-10-0-139-237 kubenswrapper[2567]: E0917 17:41:07.282085    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4cb4f" podUID="6d0a7973-cc86-4fda-9056-11b5923bd14c"
Sep 17 17:41:08.281338 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:08.281305    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vjstm"
Sep 17 17:41:08.281480 ip-10-0-139-237 kubenswrapper[2567]: E0917 17:41:08.281441    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vjstm" podUID="db73a8fe-d34c-41c8-8ba4-8040b4bacf13"
Sep 17 17:41:09.281456 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:09.281421    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4cb4f"
Sep 17 17:41:09.281990 ip-10-0-139-237 kubenswrapper[2567]: E0917 17:41:09.281570    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4cb4f" podUID="6d0a7973-cc86-4fda-9056-11b5923bd14c"
Sep 17 17:41:09.281990 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:09.281842    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xg2wr"
Sep 17 17:41:09.281990 ip-10-0-139-237 kubenswrapper[2567]: E0917 17:41:09.281967    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xg2wr" podUID="5ab7aae3-053b-4e36-9c4a-c523f0cfb615"
Sep 17 17:41:10.281583 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:10.281466    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vjstm"
Sep 17 17:41:10.282057 ip-10-0-139-237 kubenswrapper[2567]: E0917 17:41:10.281620    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vjstm" podUID="db73a8fe-d34c-41c8-8ba4-8040b4bacf13"
Sep 17 17:41:10.864790 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:10.864757    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/db73a8fe-d34c-41c8-8ba4-8040b4bacf13-original-pull-secret\") pod \"global-pull-secret-syncer-vjstm\" (UID: \"db73a8fe-d34c-41c8-8ba4-8040b4bacf13\") " pod="kube-system/global-pull-secret-syncer-vjstm"
Sep 17 17:41:10.864790 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:10.864796    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5ab7aae3-053b-4e36-9c4a-c523f0cfb615-metrics-certs\") pod \"network-metrics-daemon-xg2wr\" (UID: \"5ab7aae3-053b-4e36-9c4a-c523f0cfb615\") " pod="openshift-multus/network-metrics-daemon-xg2wr"
Sep 17 17:41:10.865035 ip-10-0-139-237 kubenswrapper[2567]: E0917 17:41:10.864935    2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:41:10.865035 ip-10-0-139-237 kubenswrapper[2567]: E0917 17:41:10.865026    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5ab7aae3-053b-4e36-9c4a-c523f0cfb615-metrics-certs podName:5ab7aae3-053b-4e36-9c4a-c523f0cfb615 nodeName:}" failed. No retries permitted until 2026-09-17 17:41:18.864969709 +0000 UTC m=+19.888574838 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5ab7aae3-053b-4e36-9c4a-c523f0cfb615-metrics-certs") pod "network-metrics-daemon-xg2wr" (UID: "5ab7aae3-053b-4e36-9c4a-c523f0cfb615") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:41:10.865436 ip-10-0-139-237 kubenswrapper[2567]: E0917 17:41:10.865341    2567 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 17:41:10.865436 ip-10-0-139-237 kubenswrapper[2567]: E0917 17:41:10.865412    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/db73a8fe-d34c-41c8-8ba4-8040b4bacf13-original-pull-secret podName:db73a8fe-d34c-41c8-8ba4-8040b4bacf13 nodeName:}" failed. No retries permitted until 2026-09-17 17:41:18.865394684 +0000 UTC m=+19.888999801 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/db73a8fe-d34c-41c8-8ba4-8040b4bacf13-original-pull-secret") pod "global-pull-secret-syncer-vjstm" (UID: "db73a8fe-d34c-41c8-8ba4-8040b4bacf13") : object "kube-system"/"original-pull-secret" not registered
Sep 17 17:41:11.066255 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:11.066121    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lndx7\" (UniqueName: \"kubernetes.io/projected/6d0a7973-cc86-4fda-9056-11b5923bd14c-kube-api-access-lndx7\") pod \"network-check-target-4cb4f\" (UID: \"6d0a7973-cc86-4fda-9056-11b5923bd14c\") " pod="openshift-network-diagnostics/network-check-target-4cb4f"
Sep 17 17:41:11.066422 ip-10-0-139-237 kubenswrapper[2567]: E0917 17:41:11.066312    2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 17:41:11.066422 ip-10-0-139-237 kubenswrapper[2567]: E0917 17:41:11.066341    2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 17:41:11.066422 ip-10-0-139-237 kubenswrapper[2567]: E0917 17:41:11.066355    2567 projected.go:196] Error preparing data for projected volume kube-api-access-lndx7 for pod openshift-network-diagnostics/network-check-target-4cb4f: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:41:11.066422 ip-10-0-139-237 kubenswrapper[2567]: E0917 17:41:11.066423    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6d0a7973-cc86-4fda-9056-11b5923bd14c-kube-api-access-lndx7 podName:6d0a7973-cc86-4fda-9056-11b5923bd14c nodeName:}" failed. No retries permitted until 2026-09-17 17:41:19.06640349 +0000 UTC m=+20.090008609 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-lndx7" (UniqueName: "kubernetes.io/projected/6d0a7973-cc86-4fda-9056-11b5923bd14c-kube-api-access-lndx7") pod "network-check-target-4cb4f" (UID: "6d0a7973-cc86-4fda-9056-11b5923bd14c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:41:11.281887 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:11.281359    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xg2wr"
Sep 17 17:41:11.281887 ip-10-0-139-237 kubenswrapper[2567]: E0917 17:41:11.281441    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xg2wr" podUID="5ab7aae3-053b-4e36-9c4a-c523f0cfb615"
Sep 17 17:41:11.281887 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:11.281708    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4cb4f"
Sep 17 17:41:11.281887 ip-10-0-139-237 kubenswrapper[2567]: E0917 17:41:11.281766    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4cb4f" podUID="6d0a7973-cc86-4fda-9056-11b5923bd14c"
Sep 17 17:41:12.280934 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:12.280899    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vjstm"
Sep 17 17:41:12.281112 ip-10-0-139-237 kubenswrapper[2567]: E0917 17:41:12.281030    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vjstm" podUID="db73a8fe-d34c-41c8-8ba4-8040b4bacf13"
Sep 17 17:41:13.280907 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:13.280873    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4cb4f"
Sep 17 17:41:13.281479 ip-10-0-139-237 kubenswrapper[2567]: E0917 17:41:13.281003    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4cb4f" podUID="6d0a7973-cc86-4fda-9056-11b5923bd14c"
Sep 17 17:41:13.281479 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:13.281068    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xg2wr"
Sep 17 17:41:13.281479 ip-10-0-139-237 kubenswrapper[2567]: E0917 17:41:13.281204    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xg2wr" podUID="5ab7aae3-053b-4e36-9c4a-c523f0cfb615"
Sep 17 17:41:14.280949 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:14.280915    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vjstm"
Sep 17 17:41:14.281297 ip-10-0-139-237 kubenswrapper[2567]: E0917 17:41:14.281035    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vjstm" podUID="db73a8fe-d34c-41c8-8ba4-8040b4bacf13"
Sep 17 17:41:15.281447 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:15.281417    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4cb4f"
Sep 17 17:41:15.281882 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:15.281406    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xg2wr"
Sep 17 17:41:15.281882 ip-10-0-139-237 kubenswrapper[2567]: E0917 17:41:15.281636    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4cb4f" podUID="6d0a7973-cc86-4fda-9056-11b5923bd14c"
Sep 17 17:41:15.281882 ip-10-0-139-237 kubenswrapper[2567]: E0917 17:41:15.281669    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xg2wr" podUID="5ab7aae3-053b-4e36-9c4a-c523f0cfb615"
Sep 17 17:41:16.280658 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:16.280629    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vjstm"
Sep 17 17:41:16.280835 ip-10-0-139-237 kubenswrapper[2567]: E0917 17:41:16.280749    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vjstm" podUID="db73a8fe-d34c-41c8-8ba4-8040b4bacf13"
Sep 17 17:41:17.280860 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:17.280830    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4cb4f"
Sep 17 17:41:17.281284 ip-10-0-139-237 kubenswrapper[2567]: E0917 17:41:17.280926    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4cb4f" podUID="6d0a7973-cc86-4fda-9056-11b5923bd14c"
Sep 17 17:41:17.281284 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:17.280992    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xg2wr"
Sep 17 17:41:17.281284 ip-10-0-139-237 kubenswrapper[2567]: E0917 17:41:17.281113    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xg2wr" podUID="5ab7aae3-053b-4e36-9c4a-c523f0cfb615"
Sep 17 17:41:18.280676 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:18.280638    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vjstm"
Sep 17 17:41:18.280862 ip-10-0-139-237 kubenswrapper[2567]: E0917 17:41:18.280754    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vjstm" podUID="db73a8fe-d34c-41c8-8ba4-8040b4bacf13"
Sep 17 17:41:18.922665 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:18.922632    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/db73a8fe-d34c-41c8-8ba4-8040b4bacf13-original-pull-secret\") pod \"global-pull-secret-syncer-vjstm\" (UID: \"db73a8fe-d34c-41c8-8ba4-8040b4bacf13\") " pod="kube-system/global-pull-secret-syncer-vjstm"
Sep 17 17:41:18.922665 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:18.922671    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5ab7aae3-053b-4e36-9c4a-c523f0cfb615-metrics-certs\") pod \"network-metrics-daemon-xg2wr\" (UID: \"5ab7aae3-053b-4e36-9c4a-c523f0cfb615\") " pod="openshift-multus/network-metrics-daemon-xg2wr"
Sep 17 17:41:18.923084 ip-10-0-139-237 kubenswrapper[2567]: E0917 17:41:18.922774    2567 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 17:41:18.923084 ip-10-0-139-237 kubenswrapper[2567]: E0917 17:41:18.922805    2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:41:18.923084 ip-10-0-139-237 kubenswrapper[2567]: E0917 17:41:18.922835    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/db73a8fe-d34c-41c8-8ba4-8040b4bacf13-original-pull-secret podName:db73a8fe-d34c-41c8-8ba4-8040b4bacf13 nodeName:}" failed. No retries permitted until 2026-09-17 17:41:34.92281902 +0000 UTC m=+35.946424139 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/db73a8fe-d34c-41c8-8ba4-8040b4bacf13-original-pull-secret") pod "global-pull-secret-syncer-vjstm" (UID: "db73a8fe-d34c-41c8-8ba4-8040b4bacf13") : object "kube-system"/"original-pull-secret" not registered
Sep 17 17:41:18.923084 ip-10-0-139-237 kubenswrapper[2567]: E0917 17:41:18.922849    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5ab7aae3-053b-4e36-9c4a-c523f0cfb615-metrics-certs podName:5ab7aae3-053b-4e36-9c4a-c523f0cfb615 nodeName:}" failed. No retries permitted until 2026-09-17 17:41:34.922842472 +0000 UTC m=+35.946447589 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5ab7aae3-053b-4e36-9c4a-c523f0cfb615-metrics-certs") pod "network-metrics-daemon-xg2wr" (UID: "5ab7aae3-053b-4e36-9c4a-c523f0cfb615") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:41:19.124589 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:19.124551    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lndx7\" (UniqueName: \"kubernetes.io/projected/6d0a7973-cc86-4fda-9056-11b5923bd14c-kube-api-access-lndx7\") pod \"network-check-target-4cb4f\" (UID: \"6d0a7973-cc86-4fda-9056-11b5923bd14c\") " pod="openshift-network-diagnostics/network-check-target-4cb4f"
Sep 17 17:41:19.124758 ip-10-0-139-237 kubenswrapper[2567]: E0917 17:41:19.124728    2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 17:41:19.124758 ip-10-0-139-237 kubenswrapper[2567]: E0917 17:41:19.124749    2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 17:41:19.124758 ip-10-0-139-237 kubenswrapper[2567]: E0917 17:41:19.124759    2567 projected.go:196] Error preparing data for projected volume kube-api-access-lndx7 for pod openshift-network-diagnostics/network-check-target-4cb4f: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:41:19.124856 ip-10-0-139-237 kubenswrapper[2567]: E0917 17:41:19.124804    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6d0a7973-cc86-4fda-9056-11b5923bd14c-kube-api-access-lndx7 podName:6d0a7973-cc86-4fda-9056-11b5923bd14c nodeName:}" failed. No retries permitted until 2026-09-17 17:41:35.124791651 +0000 UTC m=+36.148396765 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-lndx7" (UniqueName: "kubernetes.io/projected/6d0a7973-cc86-4fda-9056-11b5923bd14c-kube-api-access-lndx7") pod "network-check-target-4cb4f" (UID: "6d0a7973-cc86-4fda-9056-11b5923bd14c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:41:19.281239 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:19.281158    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xg2wr"
Sep 17 17:41:19.281389 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:19.281158    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4cb4f"
Sep 17 17:41:19.281389 ip-10-0-139-237 kubenswrapper[2567]: E0917 17:41:19.281301    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xg2wr" podUID="5ab7aae3-053b-4e36-9c4a-c523f0cfb615"
Sep 17 17:41:19.281389 ip-10-0-139-237 kubenswrapper[2567]: E0917 17:41:19.281375    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4cb4f" podUID="6d0a7973-cc86-4fda-9056-11b5923bd14c"
Sep 17 17:41:20.281248 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:20.281215    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vjstm"
Sep 17 17:41:20.281695 ip-10-0-139-237 kubenswrapper[2567]: E0917 17:41:20.281329    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vjstm" podUID="db73a8fe-d34c-41c8-8ba4-8040b4bacf13"
Sep 17 17:41:21.281737 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:21.281703    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4cb4f"
Sep 17 17:41:21.282134 ip-10-0-139-237 kubenswrapper[2567]: E0917 17:41:21.281789    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4cb4f" podUID="6d0a7973-cc86-4fda-9056-11b5923bd14c"
Sep 17 17:41:21.282134 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:21.281879    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xg2wr"
Sep 17 17:41:21.282134 ip-10-0-139-237 kubenswrapper[2567]: E0917 17:41:21.281999    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xg2wr" podUID="5ab7aae3-053b-4e36-9c4a-c523f0cfb615"
Sep 17 17:41:22.281409 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:22.281217    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vjstm"
Sep 17 17:41:22.281626 ip-10-0-139-237 kubenswrapper[2567]: E0917 17:41:22.281518    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vjstm" podUID="db73a8fe-d34c-41c8-8ba4-8040b4bacf13"
Sep 17 17:41:22.464268 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:22.464185    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-dsxjr" event={"ID":"41a677ee-5541-4579-a6e9-93b915b89d2d","Type":"ContainerStarted","Data":"dc96bf0aa5247e9ebda60ff8ffc91be3997938a5e6bd7c3fc2ca2c5e0e01bab2"}
Sep 17 17:41:22.466010 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:22.465979    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-vvjl9" event={"ID":"451c115d-30f1-4679-be41-ff6d2beedaf8","Type":"ContainerStarted","Data":"c2fc7d9f8c9efe261fb5e5aa6e5805ed19940a85e9aa23bbf2dd3bd99b9ea94d"}
Sep 17 17:41:22.467866 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:22.467814    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-454w2" event={"ID":"19204c4a-e05d-4159-94c8-477036fa461a","Type":"ContainerStarted","Data":"6debeec214775d24a07858eb9cca4d4d54d788abb957db13da897c098a414192"}
Sep 17 17:41:22.469418 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:22.469387    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-j5jw8" event={"ID":"9a0f326b-5e07-4829-bc1f-084edb7f3b17","Type":"ContainerStarted","Data":"9e76eabd119dd1187edec14a96a7107bdcfc10664763dc0bbaa071eea57bbb62"}
Sep 17 17:41:22.472566 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:22.472521    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-cfn7g" event={"ID":"e45ad616-0131-4f4e-adf4-b567b76edc38","Type":"ContainerStarted","Data":"47b73c46f2c2dd18858405d74bb165b89bc016797d4b5f26c2d6420438d1ab97"}
Sep 17 17:41:22.474292 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:22.474272    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6xdld" event={"ID":"7e56ce66-8c6f-4fa0-9560-6210a25593fa","Type":"ContainerStarted","Data":"badc50903d5457942300ebaa010ef260323efa02beba17f4ce7478151613230d"}
Sep 17 17:41:22.476010 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:22.475987    2567 generic.go:356] "Generic (PLEG): container finished" podID="bb03de28-d033-4586-8a80-54465decf01e" containerID="ffbcd213a8562b5e43d9beca1fe0c664085f655b77a28eafe28dcf02a1a6667a" exitCode=0
Sep 17 17:41:22.476105 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:22.476066    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ptjgb" event={"ID":"bb03de28-d033-4586-8a80-54465decf01e","Type":"ContainerDied","Data":"ffbcd213a8562b5e43d9beca1fe0c664085f655b77a28eafe28dcf02a1a6667a"}
Sep 17 17:41:22.480073 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:22.480053    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vwtvz" event={"ID":"0a5f3756-68b8-4e1d-97fc-75a9e6e6fb27","Type":"ContainerStarted","Data":"51bf5ae26cb9a901fabb993a07690107003d2bd54b8ef9cd3301be7edf2717cc"}
Sep 17 17:41:22.480192 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:22.480111    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vwtvz" event={"ID":"0a5f3756-68b8-4e1d-97fc-75a9e6e6fb27","Type":"ContainerStarted","Data":"52f14b3bcd4a13570583baee128501f9d5ff0dac1d5915daddbd115e25300c68"}
Sep 17 17:41:22.480192 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:22.480145    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vwtvz" event={"ID":"0a5f3756-68b8-4e1d-97fc-75a9e6e6fb27","Type":"ContainerStarted","Data":"871dc0ea2f1b0bf8f2e36010b78e3c1491ebca516b1ec541a208c2d7ca3570c7"}
Sep 17 17:41:22.480192 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:22.480187    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vwtvz" event={"ID":"0a5f3756-68b8-4e1d-97fc-75a9e6e6fb27","Type":"ContainerStarted","Data":"92d4e347db40ba66032e2cc1df685bf75c8af1c1f00796ee7487b29a37ed2426"}
Sep 17 17:41:22.480292 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:22.480225    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vwtvz" event={"ID":"0a5f3756-68b8-4e1d-97fc-75a9e6e6fb27","Type":"ContainerStarted","Data":"747a2ddf1d2f96f8ded72d3bb024d15fab6cefbb1800d697f5a02ded1735b9cc"}
Sep 17 17:41:22.480292 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:22.480253    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vwtvz" event={"ID":"0a5f3756-68b8-4e1d-97fc-75a9e6e6fb27","Type":"ContainerStarted","Data":"4bd3d8f21519bab6c15285c6386299aecec93b018a0be1817296e6792b535a0b"}
Sep 17 17:41:22.483367 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:22.483326    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-dsxjr" podStartSLOduration=4.15358813 podStartE2EDuration="21.483310165s" totalImagesPullingTime="17.329722035s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:41:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:41:22.482459606 +0000 UTC m=+23.506064761" watchObservedRunningTime="2026-09-17 17:41:22.483310165 +0000 UTC m=+23.506915293"
Sep 17 17:41:22.500034 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:22.499989    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-vvjl9" podStartSLOduration=4.228714157 podStartE2EDuration="21.499977403s" totalImagesPullingTime="17.271263246s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:41:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:41:22.499562294 +0000 UTC m=+23.523167442" watchObservedRunningTime="2026-09-17 17:41:22.499977403 +0000 UTC m=+23.523582538"
Sep 17 17:41:22.518894 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:22.518251    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-454w2" podStartSLOduration=4.32297848 podStartE2EDuration="21.518235657s" totalImagesPullingTime="17.195257177s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:41:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:41:22.518028947 +0000 UTC m=+23.541634085" watchObservedRunningTime="2026-09-17 17:41:22.518235657 +0000 UTC m=+23.541840794"
Sep 17 17:41:22.539437 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:22.539394    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-j5jw8" podStartSLOduration=11.649787797 podStartE2EDuration="21.53938137s" totalImagesPullingTime="9.889593573s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:41:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:41:22.539071975 +0000 UTC m=+23.562677111" watchObservedRunningTime="2026-09-17 17:41:22.53938137 +0000 UTC m=+23.562986505"
Sep 17 17:41:22.643054 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:22.642852    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:22.659657 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:22.659623    2567 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 17 17:41:22.733467 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:22.733369    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:22.889651 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:22.889622    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-j5jw8"
Sep 17 17:41:22.890205 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:22.890184    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-j5jw8"
Sep 17 17:41:22.908468 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:22.908425    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-cfn7g" podStartSLOduration=4.717938123 podStartE2EDuration="21.90841082s" totalImagesPullingTime="17.190472697s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:41:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:41:22.589334054 +0000 UTC m=+23.612939212" watchObservedRunningTime="2026-09-17 17:41:22.90841082 +0000 UTC m=+23.932015955"
Sep 17 17:41:23.178881 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:23.178810    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-cfn7g_e45ad616-0131-4f4e-adf4-b567b76edc38/node-ca/0.log"
Sep 17 17:41:23.265573 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:23.265446    2567 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-17T17:41:22.659643195Z","UUID":"6e667c05-a474-4089-8207-536a62c59de9","Handler":null,"Name":"","Endpoint":""}
Sep 17 17:41:23.267484 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:23.267460    2567 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 17 17:41:23.267612 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:23.267500    2567 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 17 17:41:23.280921 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:23.280898    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4cb4f"
Sep 17 17:41:23.281033 ip-10-0-139-237 kubenswrapper[2567]: E0917 17:41:23.281014    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4cb4f" podUID="6d0a7973-cc86-4fda-9056-11b5923bd14c"
Sep 17 17:41:23.281086 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:23.281073    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xg2wr"
Sep 17 17:41:23.281177 ip-10-0-139-237 kubenswrapper[2567]: E0917 17:41:23.281162    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xg2wr" podUID="5ab7aae3-053b-4e36-9c4a-c523f0cfb615"
Sep 17 17:41:23.321545 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:23.321500    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:23.383282 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:23.383239    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:23.484459 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:23.484386    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6xdld" event={"ID":"7e56ce66-8c6f-4fa0-9560-6210a25593fa","Type":"ContainerStarted","Data":"00dd4af9f2fec414b8f91e967d11e10e3c40d80005a2a23a1ff1f9ccfe596e99"}
Sep 17 17:41:23.486241 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:23.486198    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-zrbgv" event={"ID":"00f9603c-1639-4399-92ae-c1e4bf354395","Type":"ContainerStarted","Data":"60a24051db42d005c919d8afa7855494982e5cf9f97575d3d7af5c4a6d95a810"}
Sep 17 17:41:23.505019 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:23.504969    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-zrbgv" podStartSLOduration=5.236907463 podStartE2EDuration="22.50495107s" totalImagesPullingTime="17.268043607s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:41:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:41:23.504288743 +0000 UTC m=+24.527893881" watchObservedRunningTime="2026-09-17 17:41:23.50495107 +0000 UTC m=+24.528556206"
Sep 17 17:41:23.628059 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:23.628028    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:23.696368 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:23.696325    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:24.199283 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:24.199236    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:24.281752 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:24.281720    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vjstm"
Sep 17 17:41:24.281948 ip-10-0-139-237 kubenswrapper[2567]: E0917 17:41:24.281879    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vjstm" podUID="db73a8fe-d34c-41c8-8ba4-8040b4bacf13"
Sep 17 17:41:24.299645 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:24.299612    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:24.399660 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:24.399557    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:24.491312 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:24.491274    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6xdld" event={"ID":"7e56ce66-8c6f-4fa0-9560-6210a25593fa","Type":"ContainerStarted","Data":"e695edf3b6a17fda67f0bbc3b4bf651eca1959ca3bb39f2792b83dfa03311e17"}
Sep 17 17:41:24.495193 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:24.495163    2567 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 17 17:41:24.495330 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:24.495191    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vwtvz" event={"ID":"0a5f3756-68b8-4e1d-97fc-75a9e6e6fb27","Type":"ContainerStarted","Data":"f37c2e3ccaa2d58d0a043f7a5784905954661871c041558a0fef32c988e5ddbf"}
Sep 17 17:41:24.514606 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:24.514548    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6xdld" podStartSLOduration=4.74400774 podStartE2EDuration="23.514514149s" totalImagesPullingTime="18.770506409s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:41:01 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:41:24.514425621 +0000 UTC m=+25.538030759" watchObservedRunningTime="2026-09-17 17:41:24.514514149 +0000 UTC m=+25.538119287"
Sep 17 17:41:24.599961 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:24.599913    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:24.664962 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:24.664867    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:24.731484 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:24.731450    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:25.280856 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:25.280822    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4cb4f"
Sep 17 17:41:25.281021 ip-10-0-139-237 kubenswrapper[2567]: E0917 17:41:25.280935    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4cb4f" podUID="6d0a7973-cc86-4fda-9056-11b5923bd14c"
Sep 17 17:41:25.281021 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:25.280988    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xg2wr"
Sep 17 17:41:25.281146 ip-10-0-139-237 kubenswrapper[2567]: E0917 17:41:25.281118    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xg2wr" podUID="5ab7aae3-053b-4e36-9c4a-c523f0cfb615"
Sep 17 17:41:25.446781 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:25.446743    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:25.707096 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:25.707013    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:25.830915 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:25.830886    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:26.052895 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:26.052812    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:26.145897 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:26.145865    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:26.281374 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:26.281347    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vjstm"
Sep 17 17:41:26.281546 ip-10-0-139-237 kubenswrapper[2567]: E0917 17:41:26.281451    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vjstm" podUID="db73a8fe-d34c-41c8-8ba4-8040b4bacf13"
Sep 17 17:41:26.369602 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:26.369518    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:26.587282 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:26.586899    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:27.281583 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:27.281551    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4cb4f"
Sep 17 17:41:27.281583 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:27.281584    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xg2wr"
Sep 17 17:41:27.282392 ip-10-0-139-237 kubenswrapper[2567]: E0917 17:41:27.281665    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4cb4f" podUID="6d0a7973-cc86-4fda-9056-11b5923bd14c"
Sep 17 17:41:27.282392 ip-10-0-139-237 kubenswrapper[2567]: E0917 17:41:27.281783    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xg2wr" podUID="5ab7aae3-053b-4e36-9c4a-c523f0cfb615"
Sep 17 17:41:27.355928 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:27.355893    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:27.505727 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:27.505692    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vwtvz" event={"ID":"0a5f3756-68b8-4e1d-97fc-75a9e6e6fb27","Type":"ContainerStarted","Data":"e47d1c3a136812da3c01acacad2492f15352645251adf28d62802b1fae7612f9"}
Sep 17 17:41:27.505948 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:27.505924    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-vwtvz"
Sep 17 17:41:27.507606 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:27.507583    2567 generic.go:356] "Generic (PLEG): container finished" podID="bb03de28-d033-4586-8a80-54465decf01e" containerID="d9a7ee22396ccc09791f565969f2cecc7d0176629e8ecaaf3771b7163945a5a7" exitCode=0
Sep 17 17:41:27.507680 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:27.507612    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ptjgb" event={"ID":"bb03de28-d033-4586-8a80-54465decf01e","Type":"ContainerDied","Data":"d9a7ee22396ccc09791f565969f2cecc7d0176629e8ecaaf3771b7163945a5a7"}
Sep 17 17:41:27.521605 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:27.521584    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-vwtvz"
Sep 17 17:41:27.540097 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:27.540057    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-vwtvz" podStartSLOduration=9.257950002 podStartE2EDuration="26.540045107s" totalImagesPullingTime="17.282095105s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:41:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:41:27.539752645 +0000 UTC m=+28.563357800" watchObservedRunningTime="2026-09-17 17:41:27.540045107 +0000 UTC m=+28.563650270"
Sep 17 17:41:27.638036 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:27.638004    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:27.941787 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:27.941701    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:28.060227 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:28.060195    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:28.161463 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:28.161426    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:28.280684 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:28.280621    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vjstm"
Sep 17 17:41:28.280825 ip-10-0-139-237 kubenswrapper[2567]: E0917 17:41:28.280706    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vjstm" podUID="db73a8fe-d34c-41c8-8ba4-8040b4bacf13"
Sep 17 17:41:28.343376 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:28.343344    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:28.509760 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:28.509730    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-vwtvz"
Sep 17 17:41:28.509924 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:28.509798    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-vwtvz"
Sep 17 17:41:28.523807 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:28.523787    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-vwtvz"
Sep 17 17:41:28.748608 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:28.748578    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:29.200190 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:29.200158    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:29.281706 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:29.281680    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4cb4f"
Sep 17 17:41:29.281875 ip-10-0-139-237 kubenswrapper[2567]: E0917 17:41:29.281795    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4cb4f" podUID="6d0a7973-cc86-4fda-9056-11b5923bd14c"
Sep 17 17:41:29.281875 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:29.281851    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xg2wr"
Sep 17 17:41:29.281962 ip-10-0-139-237 kubenswrapper[2567]: E0917 17:41:29.281951    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xg2wr" podUID="5ab7aae3-053b-4e36-9c4a-c523f0cfb615"
Sep 17 17:41:29.331946 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:29.331916    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:29.513085 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:29.513010    2567 generic.go:356] "Generic (PLEG): container finished" podID="bb03de28-d033-4586-8a80-54465decf01e" containerID="38a678d92d04c48bc79fafa7ccfb11d34aab0875649d83ec290c8d5782920c07" exitCode=0
Sep 17 17:41:29.513635 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:29.513089    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ptjgb" event={"ID":"bb03de28-d033-4586-8a80-54465decf01e","Type":"ContainerDied","Data":"38a678d92d04c48bc79fafa7ccfb11d34aab0875649d83ec290c8d5782920c07"}
Sep 17 17:41:29.549402 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:29.549377    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:29.756661 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:29.756629    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:29.944067 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:29.944025    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:30.280883 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:30.280817    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vjstm"
Sep 17 17:41:30.281030 ip-10-0-139-237 kubenswrapper[2567]: E0917 17:41:30.280933    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vjstm" podUID="db73a8fe-d34c-41c8-8ba4-8040b4bacf13"
Sep 17 17:41:30.357545 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:30.357493    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:30.964092 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:30.964014    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:31.281919 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:31.281836    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4cb4f"
Sep 17 17:41:31.281919 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:31.281873    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xg2wr"
Sep 17 17:41:31.282139 ip-10-0-139-237 kubenswrapper[2567]: E0917 17:41:31.281951    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4cb4f" podUID="6d0a7973-cc86-4fda-9056-11b5923bd14c"
Sep 17 17:41:31.282139 ip-10-0-139-237 kubenswrapper[2567]: E0917 17:41:31.282057    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xg2wr" podUID="5ab7aae3-053b-4e36-9c4a-c523f0cfb615"
Sep 17 17:41:31.518540 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:31.518503    2567 generic.go:356] "Generic (PLEG): container finished" podID="bb03de28-d033-4586-8a80-54465decf01e" containerID="0d1e428b67a8e56b9622c3291901163db25b4aebade3eac92255bac6ea42032f" exitCode=0
Sep 17 17:41:31.518708 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:31.518594    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ptjgb" event={"ID":"bb03de28-d033-4586-8a80-54465decf01e","Type":"ContainerDied","Data":"0d1e428b67a8e56b9622c3291901163db25b4aebade3eac92255bac6ea42032f"}
Sep 17 17:41:31.553058 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:31.553033    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:31.659770 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:31.659743    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:31.761421 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:31.761393    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:32.162943 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:32.162912    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:32.237401 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:32.237365    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:32.280764 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:32.280737    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vjstm"
Sep 17 17:41:32.280933 ip-10-0-139-237 kubenswrapper[2567]: E0917 17:41:32.280849    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vjstm" podUID="db73a8fe-d34c-41c8-8ba4-8040b4bacf13"
Sep 17 17:41:32.347944 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:32.347918    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:32.971100 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:32.971067    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:33.281293 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:33.281207    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4cb4f"
Sep 17 17:41:33.281768 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:33.281207    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xg2wr"
Sep 17 17:41:33.281768 ip-10-0-139-237 kubenswrapper[2567]: E0917 17:41:33.281340    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4cb4f" podUID="6d0a7973-cc86-4fda-9056-11b5923bd14c"
Sep 17 17:41:33.281768 ip-10-0-139-237 kubenswrapper[2567]: E0917 17:41:33.281400    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xg2wr" podUID="5ab7aae3-053b-4e36-9c4a-c523f0cfb615"
Sep 17 17:41:33.283982 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:33.283963    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-j5jw8"
Sep 17 17:41:33.284102 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:33.284045    2567 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 17 17:41:33.284446 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:33.284429    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-j5jw8"
Sep 17 17:41:33.367288 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:33.367261    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:33.553389 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:33.553356    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:33.945505 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:33.945423    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:34.281689 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:34.281428    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vjstm"
Sep 17 17:41:34.282139 ip-10-0-139-237 kubenswrapper[2567]: E0917 17:41:34.281724    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vjstm" podUID="db73a8fe-d34c-41c8-8ba4-8040b4bacf13"
Sep 17 17:41:34.426910 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:34.426856    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:34.488070 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:34.488038    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:34.560157 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:34.560126    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:34.943690 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:34.943617    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/db73a8fe-d34c-41c8-8ba4-8040b4bacf13-original-pull-secret\") pod \"global-pull-secret-syncer-vjstm\" (UID: \"db73a8fe-d34c-41c8-8ba4-8040b4bacf13\") " pod="kube-system/global-pull-secret-syncer-vjstm"
Sep 17 17:41:34.943690 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:34.943659    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5ab7aae3-053b-4e36-9c4a-c523f0cfb615-metrics-certs\") pod \"network-metrics-daemon-xg2wr\" (UID: \"5ab7aae3-053b-4e36-9c4a-c523f0cfb615\") " pod="openshift-multus/network-metrics-daemon-xg2wr"
Sep 17 17:41:34.943887 ip-10-0-139-237 kubenswrapper[2567]: E0917 17:41:34.943805    2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:41:34.943887 ip-10-0-139-237 kubenswrapper[2567]: E0917 17:41:34.943808    2567 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 17:41:34.943887 ip-10-0-139-237 kubenswrapper[2567]: E0917 17:41:34.943861    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5ab7aae3-053b-4e36-9c4a-c523f0cfb615-metrics-certs podName:5ab7aae3-053b-4e36-9c4a-c523f0cfb615 nodeName:}" failed. No retries permitted until 2026-09-17 17:42:06.943846998 +0000 UTC m=+67.967452113 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5ab7aae3-053b-4e36-9c4a-c523f0cfb615-metrics-certs") pod "network-metrics-daemon-xg2wr" (UID: "5ab7aae3-053b-4e36-9c4a-c523f0cfb615") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:41:34.943887 ip-10-0-139-237 kubenswrapper[2567]: E0917 17:41:34.943883    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/db73a8fe-d34c-41c8-8ba4-8040b4bacf13-original-pull-secret podName:db73a8fe-d34c-41c8-8ba4-8040b4bacf13 nodeName:}" failed. No retries permitted until 2026-09-17 17:42:06.943868546 +0000 UTC m=+67.967473659 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/db73a8fe-d34c-41c8-8ba4-8040b4bacf13-original-pull-secret") pod "global-pull-secret-syncer-vjstm" (UID: "db73a8fe-d34c-41c8-8ba4-8040b4bacf13") : object "kube-system"/"original-pull-secret" not registered
Sep 17 17:41:35.145183 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:35.145151    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lndx7\" (UniqueName: \"kubernetes.io/projected/6d0a7973-cc86-4fda-9056-11b5923bd14c-kube-api-access-lndx7\") pod \"network-check-target-4cb4f\" (UID: \"6d0a7973-cc86-4fda-9056-11b5923bd14c\") " pod="openshift-network-diagnostics/network-check-target-4cb4f"
Sep 17 17:41:35.145366 ip-10-0-139-237 kubenswrapper[2567]: E0917 17:41:35.145345    2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 17:41:35.145447 ip-10-0-139-237 kubenswrapper[2567]: E0917 17:41:35.145376    2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 17:41:35.145447 ip-10-0-139-237 kubenswrapper[2567]: E0917 17:41:35.145390    2567 projected.go:196] Error preparing data for projected volume kube-api-access-lndx7 for pod openshift-network-diagnostics/network-check-target-4cb4f: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:41:35.145563 ip-10-0-139-237 kubenswrapper[2567]: E0917 17:41:35.145454    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6d0a7973-cc86-4fda-9056-11b5923bd14c-kube-api-access-lndx7 podName:6d0a7973-cc86-4fda-9056-11b5923bd14c nodeName:}" failed. No retries permitted until 2026-09-17 17:42:07.145436446 +0000 UTC m=+68.169041563 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-lndx7" (UniqueName: "kubernetes.io/projected/6d0a7973-cc86-4fda-9056-11b5923bd14c-kube-api-access-lndx7") pod "network-check-target-4cb4f" (UID: "6d0a7973-cc86-4fda-9056-11b5923bd14c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:41:35.151565 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:35.151525    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:35.281819 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:35.281738    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4cb4f"
Sep 17 17:41:35.282232 ip-10-0-139-237 kubenswrapper[2567]: E0917 17:41:35.281870    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4cb4f" podUID="6d0a7973-cc86-4fda-9056-11b5923bd14c"
Sep 17 17:41:35.282232 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:35.281898    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xg2wr"
Sep 17 17:41:35.282232 ip-10-0-139-237 kubenswrapper[2567]: E0917 17:41:35.282022    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xg2wr" podUID="5ab7aae3-053b-4e36-9c4a-c523f0cfb615"
Sep 17 17:41:35.344767 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:35.344736    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:35.541960 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:35.541934    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:35.762357 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:35.762320    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:36.128636 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:36.128611    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:36.229121 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:36.229097    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:36.281372 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:36.281320    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vjstm"
Sep 17 17:41:36.281480 ip-10-0-139-237 kubenswrapper[2567]: E0917 17:41:36.281433    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vjstm" podUID="db73a8fe-d34c-41c8-8ba4-8040b4bacf13"
Sep 17 17:41:36.533340 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:36.533258    2567 generic.go:356] "Generic (PLEG): container finished" podID="bb03de28-d033-4586-8a80-54465decf01e" containerID="af6df6aa4af2129258033cd3082722069f1f1de00b62497e2d17649e71a13975" exitCode=0
Sep 17 17:41:36.533340 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:36.533322    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ptjgb" event={"ID":"bb03de28-d033-4586-8a80-54465decf01e","Type":"ContainerDied","Data":"af6df6aa4af2129258033cd3082722069f1f1de00b62497e2d17649e71a13975"}
Sep 17 17:41:37.281222 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:37.281186    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4cb4f"
Sep 17 17:41:37.281407 ip-10-0-139-237 kubenswrapper[2567]: E0917 17:41:37.281287    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4cb4f" podUID="6d0a7973-cc86-4fda-9056-11b5923bd14c"
Sep 17 17:41:37.281407 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:37.281380    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xg2wr"
Sep 17 17:41:37.281495 ip-10-0-139-237 kubenswrapper[2567]: E0917 17:41:37.281466    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xg2wr" podUID="5ab7aae3-053b-4e36-9c4a-c523f0cfb615"
Sep 17 17:41:37.538719 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:37.538695    2567 generic.go:356] "Generic (PLEG): container finished" podID="bb03de28-d033-4586-8a80-54465decf01e" containerID="5ca5c75d60527acff9f67845005091491a6b041e079745302b59c0a84b541a7b" exitCode=0
Sep 17 17:41:37.539133 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:37.538745    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ptjgb" event={"ID":"bb03de28-d033-4586-8a80-54465decf01e","Type":"ContainerDied","Data":"5ca5c75d60527acff9f67845005091491a6b041e079745302b59c0a84b541a7b"}
Sep 17 17:41:38.281382 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:38.281355    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vjstm"
Sep 17 17:41:38.281589 ip-10-0-139-237 kubenswrapper[2567]: E0917 17:41:38.281447    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vjstm" podUID="db73a8fe-d34c-41c8-8ba4-8040b4bacf13"
Sep 17 17:41:38.543852 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:38.543823    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ptjgb" event={"ID":"bb03de28-d033-4586-8a80-54465decf01e","Type":"ContainerStarted","Data":"9fd1b3aa1ab53d46575fcee7f71be353f7cb414392e0df01fb0af61ba0c87780"}
Sep 17 17:41:38.569142 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:38.569106    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-ptjgb" podStartSLOduration=9.702777525 podStartE2EDuration="37.569090863s" totalImagesPullingTime="27.866313338s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:41:01 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:41:38.567915167 +0000 UTC m=+39.591520302" watchObservedRunningTime="2026-09-17 17:41:38.569090863 +0000 UTC m=+39.592695998"
Sep 17 17:41:39.281105 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:39.281076    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4cb4f"
Sep 17 17:41:39.281264 ip-10-0-139-237 kubenswrapper[2567]: E0917 17:41:39.281173    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4cb4f" podUID="6d0a7973-cc86-4fda-9056-11b5923bd14c"
Sep 17 17:41:39.281310 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:39.281258    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xg2wr"
Sep 17 17:41:39.281400 ip-10-0-139-237 kubenswrapper[2567]: E0917 17:41:39.281380    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xg2wr" podUID="5ab7aae3-053b-4e36-9c4a-c523f0cfb615"
Sep 17 17:41:40.281687 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:40.281654    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vjstm"
Sep 17 17:41:40.282055 ip-10-0-139-237 kubenswrapper[2567]: E0917 17:41:40.281754    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vjstm" podUID="db73a8fe-d34c-41c8-8ba4-8040b4bacf13"
Sep 17 17:41:41.282262 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:41.282232    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4cb4f"
Sep 17 17:41:41.282689 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:41.282302    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xg2wr"
Sep 17 17:41:41.282689 ip-10-0-139-237 kubenswrapper[2567]: E0917 17:41:41.282308    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4cb4f" podUID="6d0a7973-cc86-4fda-9056-11b5923bd14c"
Sep 17 17:41:41.282689 ip-10-0-139-237 kubenswrapper[2567]: E0917 17:41:41.282422    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xg2wr" podUID="5ab7aae3-053b-4e36-9c4a-c523f0cfb615"
Sep 17 17:41:42.280952 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:42.280912    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vjstm"
Sep 17 17:41:42.281130 ip-10-0-139-237 kubenswrapper[2567]: E0917 17:41:42.281037    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vjstm" podUID="db73a8fe-d34c-41c8-8ba4-8040b4bacf13"
Sep 17 17:41:43.281522 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:43.281486    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4cb4f"
Sep 17 17:41:43.282085 ip-10-0-139-237 kubenswrapper[2567]: E0917 17:41:43.281600    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4cb4f" podUID="6d0a7973-cc86-4fda-9056-11b5923bd14c"
Sep 17 17:41:43.282085 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:43.281653    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xg2wr"
Sep 17 17:41:43.282085 ip-10-0-139-237 kubenswrapper[2567]: E0917 17:41:43.281738    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xg2wr" podUID="5ab7aae3-053b-4e36-9c4a-c523f0cfb615"
Sep 17 17:41:43.598445 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:43.598415    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:43.703162 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:43.703132    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:44.281199 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:44.281167    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vjstm"
Sep 17 17:41:44.281364 ip-10-0-139-237 kubenswrapper[2567]: E0917 17:41:44.281277    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vjstm" podUID="db73a8fe-d34c-41c8-8ba4-8040b4bacf13"
Sep 17 17:41:44.587439 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:44.587407    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-vjstm"]
Sep 17 17:41:44.587888 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:44.587636    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vjstm"
Sep 17 17:41:44.587888 ip-10-0-139-237 kubenswrapper[2567]: E0917 17:41:44.587754    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vjstm" podUID="db73a8fe-d34c-41c8-8ba4-8040b4bacf13"
Sep 17 17:41:44.591375 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:44.591347    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-xg2wr"]
Sep 17 17:41:44.591493 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:44.591449    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xg2wr"
Sep 17 17:41:44.591603 ip-10-0-139-237 kubenswrapper[2567]: E0917 17:41:44.591579    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xg2wr" podUID="5ab7aae3-053b-4e36-9c4a-c523f0cfb615"
Sep 17 17:41:44.592168 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:44.592147    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-4cb4f"]
Sep 17 17:41:44.592263 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:44.592227    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4cb4f"
Sep 17 17:41:44.592350 ip-10-0-139-237 kubenswrapper[2567]: E0917 17:41:44.592330    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4cb4f" podUID="6d0a7973-cc86-4fda-9056-11b5923bd14c"
Sep 17 17:41:45.110955 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:45.110926    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:45.172185 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:45.172156    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:46.143903 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:46.143874    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:46.219859 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:46.219831    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:46.280793 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:46.280767    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xg2wr"
Sep 17 17:41:46.280931 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:46.280768    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vjstm"
Sep 17 17:41:46.280931 ip-10-0-139-237 kubenswrapper[2567]: E0917 17:41:46.280868    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xg2wr" podUID="5ab7aae3-053b-4e36-9c4a-c523f0cfb615"
Sep 17 17:41:46.280931 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:46.280768    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4cb4f"
Sep 17 17:41:46.281024 ip-10-0-139-237 kubenswrapper[2567]: E0917 17:41:46.280931    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vjstm" podUID="db73a8fe-d34c-41c8-8ba4-8040b4bacf13"
Sep 17 17:41:46.281024 ip-10-0-139-237 kubenswrapper[2567]: E0917 17:41:46.280991    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4cb4f" podUID="6d0a7973-cc86-4fda-9056-11b5923bd14c"
Sep 17 17:41:46.322721 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:46.322695    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:47.178194 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:47.178163    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:47.365900 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:47.365871    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:47.508181 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:47.508102    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:47.608161 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:47.608134    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:47.681395 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:47.681371    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:47.812000 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:47.811969    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:48.281175 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:48.280954    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4cb4f"
Sep 17 17:41:48.281496 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:48.280954    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vjstm"
Sep 17 17:41:48.281496 ip-10-0-139-237 kubenswrapper[2567]: E0917 17:41:48.281217    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4cb4f" podUID="6d0a7973-cc86-4fda-9056-11b5923bd14c"
Sep 17 17:41:48.281496 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:48.280954    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xg2wr"
Sep 17 17:41:48.281496 ip-10-0-139-237 kubenswrapper[2567]: E0917 17:41:48.281284    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vjstm" podUID="db73a8fe-d34c-41c8-8ba4-8040b4bacf13"
Sep 17 17:41:48.281496 ip-10-0-139-237 kubenswrapper[2567]: E0917 17:41:48.281359    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xg2wr" podUID="5ab7aae3-053b-4e36-9c4a-c523f0cfb615"
Sep 17 17:41:48.588918 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:48.588889    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:48.690471 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:48.690444    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:49.064766 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:49.064734    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:49.162973 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:49.162947    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:49.256312 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:49.256287    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-237.ec2.internal" event="NodeReady"
Sep 17 17:41:49.256449 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:49.256405    2567 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 17 17:41:49.348417 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:49.348342    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-mdbmq"]
Sep 17 17:41:49.350028 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:49.350007    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-mdbmq"
Sep 17 17:41:49.352868 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:49.352845    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/12911f3d-f3ef-4315-89d2-5760e9d6e938-tmp-dir\") pod \"dns-default-mdbmq\" (UID: \"12911f3d-f3ef-4315-89d2-5760e9d6e938\") " pod="openshift-dns/dns-default-mdbmq"
Sep 17 17:41:49.352959 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:49.352880    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sndpj\" (UniqueName: \"kubernetes.io/projected/12911f3d-f3ef-4315-89d2-5760e9d6e938-kube-api-access-sndpj\") pod \"dns-default-mdbmq\" (UID: \"12911f3d-f3ef-4315-89d2-5760e9d6e938\") " pod="openshift-dns/dns-default-mdbmq"
Sep 17 17:41:49.352997 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:49.352981    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/12911f3d-f3ef-4315-89d2-5760e9d6e938-config-volume\") pod \"dns-default-mdbmq\" (UID: \"12911f3d-f3ef-4315-89d2-5760e9d6e938\") " pod="openshift-dns/dns-default-mdbmq"
Sep 17 17:41:49.353039 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:49.353024    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/12911f3d-f3ef-4315-89d2-5760e9d6e938-metrics-tls\") pod \"dns-default-mdbmq\" (UID: \"12911f3d-f3ef-4315-89d2-5760e9d6e938\") " pod="openshift-dns/dns-default-mdbmq"
Sep 17 17:41:49.355900 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:49.355878    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-4bpm9"]
Sep 17 17:41:49.357785 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:49.357769    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-4bpm9"
Sep 17 17:41:49.360569 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:49.360548    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 17 17:41:49.360818 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:49.360802    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-gsgx7\""
Sep 17 17:41:49.361395 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:49.361379    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 17 17:41:49.369604 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:49.369589    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 17 17:41:49.369803 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:49.369784    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 17 17:41:49.369951 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:49.369909    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 17 17:41:49.370031 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:49.370002    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 17 17:41:49.370197 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:49.370182    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-b7fxl\""
Sep 17 17:41:49.387280 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:49.387256    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-mdbmq"]
Sep 17 17:41:49.388754 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:49.388731    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-4bpm9"]
Sep 17 17:41:49.397204 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:49.397185    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-b9hsk"]
Sep 17 17:41:49.398761 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:49.398747    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-b9hsk"
Sep 17 17:41:49.401580 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:49.401561    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 17 17:41:49.401782 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:49.401767    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 17 17:41:49.401894 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:49.401874    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 17 17:41:49.401993 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:49.401979    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-tzqnk\""
Sep 17 17:41:49.413939 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:49.413918    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-b9hsk"]
Sep 17 17:41:49.453871 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:49.453844    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/12911f3d-f3ef-4315-89d2-5760e9d6e938-metrics-tls\") pod \"dns-default-mdbmq\" (UID: \"12911f3d-f3ef-4315-89d2-5760e9d6e938\") " pod="openshift-dns/dns-default-mdbmq"
Sep 17 17:41:49.453954 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:49.453880    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lgnpt\" (UniqueName: \"kubernetes.io/projected/ad8f026a-99c1-4153-a25f-e4638d09d341-kube-api-access-lgnpt\") pod \"insights-runtime-extractor-4bpm9\" (UID: \"ad8f026a-99c1-4153-a25f-e4638d09d341\") " pod="openshift-insights/insights-runtime-extractor-4bpm9"
Sep 17 17:41:49.453954 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:49.453909    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ad8f026a-99c1-4153-a25f-e4638d09d341-crio-socket\") pod \"insights-runtime-extractor-4bpm9\" (UID: \"ad8f026a-99c1-4153-a25f-e4638d09d341\") " pod="openshift-insights/insights-runtime-extractor-4bpm9"
Sep 17 17:41:49.453954 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:49.453928    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ad8f026a-99c1-4153-a25f-e4638d09d341-data-volume\") pod \"insights-runtime-extractor-4bpm9\" (UID: \"ad8f026a-99c1-4153-a25f-e4638d09d341\") " pod="openshift-insights/insights-runtime-extractor-4bpm9"
Sep 17 17:41:49.454137 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:49.453986    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ad8f026a-99c1-4153-a25f-e4638d09d341-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4bpm9\" (UID: \"ad8f026a-99c1-4153-a25f-e4638d09d341\") " pod="openshift-insights/insights-runtime-extractor-4bpm9"
Sep 17 17:41:49.454137 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:49.454055    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ad8f026a-99c1-4153-a25f-e4638d09d341-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4bpm9\" (UID: \"ad8f026a-99c1-4153-a25f-e4638d09d341\") " pod="openshift-insights/insights-runtime-extractor-4bpm9"
Sep 17 17:41:49.454137 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:49.454091    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/12911f3d-f3ef-4315-89d2-5760e9d6e938-tmp-dir\") pod \"dns-default-mdbmq\" (UID: \"12911f3d-f3ef-4315-89d2-5760e9d6e938\") " pod="openshift-dns/dns-default-mdbmq"
Sep 17 17:41:49.454137 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:49.454118    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xjzcp\" (UniqueName: \"kubernetes.io/projected/882622eb-219e-4d57-99b8-4c763d1b57b2-kube-api-access-xjzcp\") pod \"ingress-canary-b9hsk\" (UID: \"882622eb-219e-4d57-99b8-4c763d1b57b2\") " pod="openshift-ingress-canary/ingress-canary-b9hsk"
Sep 17 17:41:49.454378 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:49.454144    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sndpj\" (UniqueName: \"kubernetes.io/projected/12911f3d-f3ef-4315-89d2-5760e9d6e938-kube-api-access-sndpj\") pod \"dns-default-mdbmq\" (UID: \"12911f3d-f3ef-4315-89d2-5760e9d6e938\") " pod="openshift-dns/dns-default-mdbmq"
Sep 17 17:41:49.454378 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:49.454184    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/882622eb-219e-4d57-99b8-4c763d1b57b2-cert\") pod \"ingress-canary-b9hsk\" (UID: \"882622eb-219e-4d57-99b8-4c763d1b57b2\") " pod="openshift-ingress-canary/ingress-canary-b9hsk"
Sep 17 17:41:49.454378 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:49.454242    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/12911f3d-f3ef-4315-89d2-5760e9d6e938-config-volume\") pod \"dns-default-mdbmq\" (UID: \"12911f3d-f3ef-4315-89d2-5760e9d6e938\") " pod="openshift-dns/dns-default-mdbmq"
Sep 17 17:41:49.454544 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:49.454448    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/12911f3d-f3ef-4315-89d2-5760e9d6e938-tmp-dir\") pod \"dns-default-mdbmq\" (UID: \"12911f3d-f3ef-4315-89d2-5760e9d6e938\") " pod="openshift-dns/dns-default-mdbmq"
Sep 17 17:41:49.454852 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:49.454831    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/12911f3d-f3ef-4315-89d2-5760e9d6e938-config-volume\") pod \"dns-default-mdbmq\" (UID: \"12911f3d-f3ef-4315-89d2-5760e9d6e938\") " pod="openshift-dns/dns-default-mdbmq"
Sep 17 17:41:49.458029 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:49.458013    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/12911f3d-f3ef-4315-89d2-5760e9d6e938-metrics-tls\") pod \"dns-default-mdbmq\" (UID: \"12911f3d-f3ef-4315-89d2-5760e9d6e938\") " pod="openshift-dns/dns-default-mdbmq"
Sep 17 17:41:49.464564 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:49.464525    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sndpj\" (UniqueName: \"kubernetes.io/projected/12911f3d-f3ef-4315-89d2-5760e9d6e938-kube-api-access-sndpj\") pod \"dns-default-mdbmq\" (UID: \"12911f3d-f3ef-4315-89d2-5760e9d6e938\") " pod="openshift-dns/dns-default-mdbmq"
Sep 17 17:41:49.504388 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:49.504333    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:49.554462 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:49.554442    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lgnpt\" (UniqueName: \"kubernetes.io/projected/ad8f026a-99c1-4153-a25f-e4638d09d341-kube-api-access-lgnpt\") pod \"insights-runtime-extractor-4bpm9\" (UID: \"ad8f026a-99c1-4153-a25f-e4638d09d341\") " pod="openshift-insights/insights-runtime-extractor-4bpm9"
Sep 17 17:41:49.554567 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:49.554474    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ad8f026a-99c1-4153-a25f-e4638d09d341-crio-socket\") pod \"insights-runtime-extractor-4bpm9\" (UID: \"ad8f026a-99c1-4153-a25f-e4638d09d341\") " pod="openshift-insights/insights-runtime-extractor-4bpm9"
Sep 17 17:41:49.554567 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:49.554489    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ad8f026a-99c1-4153-a25f-e4638d09d341-data-volume\") pod \"insights-runtime-extractor-4bpm9\" (UID: \"ad8f026a-99c1-4153-a25f-e4638d09d341\") " pod="openshift-insights/insights-runtime-extractor-4bpm9"
Sep 17 17:41:49.554567 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:49.554504    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ad8f026a-99c1-4153-a25f-e4638d09d341-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4bpm9\" (UID: \"ad8f026a-99c1-4153-a25f-e4638d09d341\") " pod="openshift-insights/insights-runtime-extractor-4bpm9"
Sep 17 17:41:49.554567 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:49.554549    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ad8f026a-99c1-4153-a25f-e4638d09d341-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4bpm9\" (UID: \"ad8f026a-99c1-4153-a25f-e4638d09d341\") " pod="openshift-insights/insights-runtime-extractor-4bpm9"
Sep 17 17:41:49.554758 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:49.554669    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xjzcp\" (UniqueName: \"kubernetes.io/projected/882622eb-219e-4d57-99b8-4c763d1b57b2-kube-api-access-xjzcp\") pod \"ingress-canary-b9hsk\" (UID: \"882622eb-219e-4d57-99b8-4c763d1b57b2\") " pod="openshift-ingress-canary/ingress-canary-b9hsk"
Sep 17 17:41:49.554758 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:49.554689    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ad8f026a-99c1-4153-a25f-e4638d09d341-crio-socket\") pod \"insights-runtime-extractor-4bpm9\" (UID: \"ad8f026a-99c1-4153-a25f-e4638d09d341\") " pod="openshift-insights/insights-runtime-extractor-4bpm9"
Sep 17 17:41:49.554758 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:49.554734    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/882622eb-219e-4d57-99b8-4c763d1b57b2-cert\") pod \"ingress-canary-b9hsk\" (UID: \"882622eb-219e-4d57-99b8-4c763d1b57b2\") " pod="openshift-ingress-canary/ingress-canary-b9hsk"
Sep 17 17:41:49.554886 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:49.554812    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ad8f026a-99c1-4153-a25f-e4638d09d341-data-volume\") pod \"insights-runtime-extractor-4bpm9\" (UID: \"ad8f026a-99c1-4153-a25f-e4638d09d341\") " pod="openshift-insights/insights-runtime-extractor-4bpm9"
Sep 17 17:41:49.555138 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:49.555119    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ad8f026a-99c1-4153-a25f-e4638d09d341-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4bpm9\" (UID: \"ad8f026a-99c1-4153-a25f-e4638d09d341\") " pod="openshift-insights/insights-runtime-extractor-4bpm9"
Sep 17 17:41:49.556594 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:49.556570    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ad8f026a-99c1-4153-a25f-e4638d09d341-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4bpm9\" (UID: \"ad8f026a-99c1-4153-a25f-e4638d09d341\") " pod="openshift-insights/insights-runtime-extractor-4bpm9"
Sep 17 17:41:49.556928 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:49.556909    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/882622eb-219e-4d57-99b8-4c763d1b57b2-cert\") pod \"ingress-canary-b9hsk\" (UID: \"882622eb-219e-4d57-99b8-4c763d1b57b2\") " pod="openshift-ingress-canary/ingress-canary-b9hsk"
Sep 17 17:41:49.563442 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:49.563420    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lgnpt\" (UniqueName: \"kubernetes.io/projected/ad8f026a-99c1-4153-a25f-e4638d09d341-kube-api-access-lgnpt\") pod \"insights-runtime-extractor-4bpm9\" (UID: \"ad8f026a-99c1-4153-a25f-e4638d09d341\") " pod="openshift-insights/insights-runtime-extractor-4bpm9"
Sep 17 17:41:49.564724 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:49.564701    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xjzcp\" (UniqueName: \"kubernetes.io/projected/882622eb-219e-4d57-99b8-4c763d1b57b2-kube-api-access-xjzcp\") pod \"ingress-canary-b9hsk\" (UID: \"882622eb-219e-4d57-99b8-4c763d1b57b2\") " pod="openshift-ingress-canary/ingress-canary-b9hsk"
Sep 17 17:41:49.580940 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:49.580917    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:49.659682 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:49.659617    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-mdbmq"
Sep 17 17:41:49.666549 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:49.666519    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-4bpm9"
Sep 17 17:41:49.707179 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:49.707147    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-b9hsk"
Sep 17 17:41:49.947710 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:49.947636    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-4bpm9"]
Sep 17 17:41:49.950306 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:49.950281    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-mdbmq"]
Sep 17 17:41:49.950838 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:49.950818    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad8f026a_99c1_4153_a25f_e4638d09d341.slice/crio-eb9b8ebf4fc8bd0ba334b15a12a15d366814d9eafbce287ab383b4f9a14cc6f7 WatchSource:0}: Error finding container eb9b8ebf4fc8bd0ba334b15a12a15d366814d9eafbce287ab383b4f9a14cc6f7: Status 404 returned error can't find the container with id eb9b8ebf4fc8bd0ba334b15a12a15d366814d9eafbce287ab383b4f9a14cc6f7
Sep 17 17:41:49.952953 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:49.952930    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12911f3d_f3ef_4315_89d2_5760e9d6e938.slice/crio-118106aef57e0d6234372622c053945743621191414b5f2d3ec199c0b84df196 WatchSource:0}: Error finding container 118106aef57e0d6234372622c053945743621191414b5f2d3ec199c0b84df196: Status 404 returned error can't find the container with id 118106aef57e0d6234372622c053945743621191414b5f2d3ec199c0b84df196
Sep 17 17:41:49.984664 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:49.984640    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-b9hsk"]
Sep 17 17:41:49.988174 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:49.988148    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod882622eb_219e_4d57_99b8_4c763d1b57b2.slice/crio-1e8f4642eee250986f3b7931de61668a598fb2cce17b254fefd1b596ac835bc4 WatchSource:0}: Error finding container 1e8f4642eee250986f3b7931de61668a598fb2cce17b254fefd1b596ac835bc4: Status 404 returned error can't find the container with id 1e8f4642eee250986f3b7931de61668a598fb2cce17b254fefd1b596ac835bc4
Sep 17 17:41:50.281836 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:50.281578    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xg2wr"
Sep 17 17:41:50.281982 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:50.281593    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vjstm"
Sep 17 17:41:50.281982 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:50.281628    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4cb4f"
Sep 17 17:41:50.286051 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:50.286024    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 17 17:41:50.286051 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:50.286041    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 17 17:41:50.286218 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:50.286089    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 17 17:41:50.286218 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:50.286131    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 17 17:41:50.286294 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:50.286234    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-lspl9\""
Sep 17 17:41:50.286421 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:50.286409    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-jc79n\""
Sep 17 17:41:50.370154 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:50.370095    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:50.472000 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:50.471633    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:50.574218 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:50.574177    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-b9hsk" event={"ID":"882622eb-219e-4d57-99b8-4c763d1b57b2","Type":"ContainerStarted","Data":"1e8f4642eee250986f3b7931de61668a598fb2cce17b254fefd1b596ac835bc4"}
Sep 17 17:41:50.575700 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:50.575670    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-mdbmq" event={"ID":"12911f3d-f3ef-4315-89d2-5760e9d6e938","Type":"ContainerStarted","Data":"118106aef57e0d6234372622c053945743621191414b5f2d3ec199c0b84df196"}
Sep 17 17:41:50.578084 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:50.577990    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-4bpm9" event={"ID":"ad8f026a-99c1-4153-a25f-e4638d09d341","Type":"ContainerStarted","Data":"215da05d463fd85e6ea5ccf3cb16f416be1bdb65238571ac4016e55c6b3b07d8"}
Sep 17 17:41:50.578084 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:50.578045    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-4bpm9" event={"ID":"ad8f026a-99c1-4153-a25f-e4638d09d341","Type":"ContainerStarted","Data":"eb9b8ebf4fc8bd0ba334b15a12a15d366814d9eafbce287ab383b4f9a14cc6f7"}
Sep 17 17:41:51.582862 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:51.582825    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-4bpm9" event={"ID":"ad8f026a-99c1-4153-a25f-e4638d09d341","Type":"ContainerStarted","Data":"941ff11964c53b78773da3ee4bf4d7baab25a0b208a4b866c83878dbb191830c"}
Sep 17 17:41:53.592066 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:53.592027    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-4bpm9" event={"ID":"ad8f026a-99c1-4153-a25f-e4638d09d341","Type":"ContainerStarted","Data":"adc1a4deaff90c3c823006642b9a7f15ec85b81566ba16f8e4cf2ec7774332b4"}
Sep 17 17:41:53.593504 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:53.593483    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-b9hsk" event={"ID":"882622eb-219e-4d57-99b8-4c763d1b57b2","Type":"ContainerStarted","Data":"430d56c68d3340078d096eec5b16edea13878fbbf20a94936bd8367c4d976667"}
Sep 17 17:41:53.595216 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:53.595194    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-mdbmq" event={"ID":"12911f3d-f3ef-4315-89d2-5760e9d6e938","Type":"ContainerStarted","Data":"e0a28a7158d7764750476dfeb3ddf355002898aaebabfccd331f70ce6127d1c6"}
Sep 17 17:41:53.595323 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:53.595231    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-mdbmq" event={"ID":"12911f3d-f3ef-4315-89d2-5760e9d6e938","Type":"ContainerStarted","Data":"1202ac8596e0d4c67f004dcbac4ea07101b71f28e26707393675ff7aee066fbc"}
Sep 17 17:41:53.595323 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:53.595307    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-mdbmq"
Sep 17 17:41:53.612651 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:53.612605    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-4bpm9" podStartSLOduration=1.389446071 podStartE2EDuration="4.612593144s" totalImagesPullingTime="3.223147073s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:41:49 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:41:53.612321463 +0000 UTC m=+54.635926599" watchObservedRunningTime="2026-09-17 17:41:53.612593144 +0000 UTC m=+54.636198280"
Sep 17 17:41:53.631788 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:53.631743    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-mdbmq" podStartSLOduration=1.28548091 podStartE2EDuration="4.631730456s" totalImagesPullingTime="3.346249546s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:41:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:41:53.631315122 +0000 UTC m=+54.654920270" watchObservedRunningTime="2026-09-17 17:41:53.631730456 +0000 UTC m=+54.655335596"
Sep 17 17:41:53.647704 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:53.647657    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-b9hsk" podStartSLOduration=1.336850712 podStartE2EDuration="4.647639342s" totalImagesPullingTime="3.31078863s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:41:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:41:53.647369488 +0000 UTC m=+54.670974624" watchObservedRunningTime="2026-09-17 17:41:53.647639342 +0000 UTC m=+54.671244479"
Sep 17 17:41:54.504121 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:54.504040    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:54.611993 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:54.611963    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:54.690392 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:54.690362    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-m66j7"]
Sep 17 17:41:54.709561 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:54.709523    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-m66j7"
Sep 17 17:41:54.716464 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:54.716444    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Sep 17 17:41:54.716571 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:54.716495    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-27nxg\""
Sep 17 17:41:54.729982 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:54.729957    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-m66j7"]
Sep 17 17:41:54.792711 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:54.792690    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:54.795722 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:54.795704    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/026b6ea8-2b9e-4f09-97a1-dc77ef61a70c-tls-certificates\") pod \"prometheus-operator-admission-webhook-5c6747dc7b-m66j7\" (UID: \"026b6ea8-2b9e-4f09-97a1-dc77ef61a70c\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-m66j7"
Sep 17 17:41:54.893476 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:54.893448    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:54.896478 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:54.896458    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/026b6ea8-2b9e-4f09-97a1-dc77ef61a70c-tls-certificates\") pod \"prometheus-operator-admission-webhook-5c6747dc7b-m66j7\" (UID: \"026b6ea8-2b9e-4f09-97a1-dc77ef61a70c\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-m66j7"
Sep 17 17:41:54.896613 ip-10-0-139-237 kubenswrapper[2567]: E0917 17:41:54.896600    2567 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-admission-webhook-tls: secret "prometheus-operator-admission-webhook-tls" not found
Sep 17 17:41:54.896673 ip-10-0-139-237 kubenswrapper[2567]: E0917 17:41:54.896664    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/026b6ea8-2b9e-4f09-97a1-dc77ef61a70c-tls-certificates podName:026b6ea8-2b9e-4f09-97a1-dc77ef61a70c nodeName:}" failed. No retries permitted until 2026-09-17 17:41:55.396650158 +0000 UTC m=+56.420255286 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certificates" (UniqueName: "kubernetes.io/secret/026b6ea8-2b9e-4f09-97a1-dc77ef61a70c-tls-certificates") pod "prometheus-operator-admission-webhook-5c6747dc7b-m66j7" (UID: "026b6ea8-2b9e-4f09-97a1-dc77ef61a70c") : secret "prometheus-operator-admission-webhook-tls" not found
Sep 17 17:41:55.220039 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:55.219952    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:55.399850 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:55.399821    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/026b6ea8-2b9e-4f09-97a1-dc77ef61a70c-tls-certificates\") pod \"prometheus-operator-admission-webhook-5c6747dc7b-m66j7\" (UID: \"026b6ea8-2b9e-4f09-97a1-dc77ef61a70c\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-m66j7"
Sep 17 17:41:55.403783 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:55.403763    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/026b6ea8-2b9e-4f09-97a1-dc77ef61a70c-tls-certificates\") pod \"prometheus-operator-admission-webhook-5c6747dc7b-m66j7\" (UID: \"026b6ea8-2b9e-4f09-97a1-dc77ef61a70c\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-m66j7"
Sep 17 17:41:55.436952 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:55.436926    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:55.536142 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:55.536078    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:55.618415 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:55.618392    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-m66j7"
Sep 17 17:41:55.822722 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:55.822691    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5fdd747cc7-gw5hv"]
Sep 17 17:41:55.855912 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:55.855888    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5fdd747cc7-gw5hv"]
Sep 17 17:41:55.856035 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:55.855983    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5fdd747cc7-gw5hv"
Sep 17 17:41:55.859171 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:55.859153    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-phvtx\""
Sep 17 17:41:55.859362 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:55.859343    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 17 17:41:55.859665 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:55.859649    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 17 17:41:55.860360 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:55.860344    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 17 17:41:55.860616 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:55.860597    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 17 17:41:55.860616 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:55.860608    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 17 17:41:55.860816 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:55.860804    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 17 17:41:55.861594 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:55.861578    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 17 17:41:55.903594 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:55.903573    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/23dbde68-e2d6-4eeb-9132-5f6c29b4d261-console-serving-cert\") pod \"console-5fdd747cc7-gw5hv\" (UID: \"23dbde68-e2d6-4eeb-9132-5f6c29b4d261\") " pod="openshift-console/console-5fdd747cc7-gw5hv"
Sep 17 17:41:55.903694 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:55.903600    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/23dbde68-e2d6-4eeb-9132-5f6c29b4d261-oauth-serving-cert\") pod \"console-5fdd747cc7-gw5hv\" (UID: \"23dbde68-e2d6-4eeb-9132-5f6c29b4d261\") " pod="openshift-console/console-5fdd747cc7-gw5hv"
Sep 17 17:41:55.903694 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:55.903616    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7zcw7\" (UniqueName: \"kubernetes.io/projected/23dbde68-e2d6-4eeb-9132-5f6c29b4d261-kube-api-access-7zcw7\") pod \"console-5fdd747cc7-gw5hv\" (UID: \"23dbde68-e2d6-4eeb-9132-5f6c29b4d261\") " pod="openshift-console/console-5fdd747cc7-gw5hv"
Sep 17 17:41:55.903694 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:55.903648    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/23dbde68-e2d6-4eeb-9132-5f6c29b4d261-console-oauth-config\") pod \"console-5fdd747cc7-gw5hv\" (UID: \"23dbde68-e2d6-4eeb-9132-5f6c29b4d261\") " pod="openshift-console/console-5fdd747cc7-gw5hv"
Sep 17 17:41:55.903694 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:55.903687    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/23dbde68-e2d6-4eeb-9132-5f6c29b4d261-console-config\") pod \"console-5fdd747cc7-gw5hv\" (UID: \"23dbde68-e2d6-4eeb-9132-5f6c29b4d261\") " pod="openshift-console/console-5fdd747cc7-gw5hv"
Sep 17 17:41:55.903823 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:55.903728    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/23dbde68-e2d6-4eeb-9132-5f6c29b4d261-service-ca\") pod \"console-5fdd747cc7-gw5hv\" (UID: \"23dbde68-e2d6-4eeb-9132-5f6c29b4d261\") " pod="openshift-console/console-5fdd747cc7-gw5hv"
Sep 17 17:41:55.917985 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:55.917965    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-m66j7"]
Sep 17 17:41:55.921289 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:55.921268    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod026b6ea8_2b9e_4f09_97a1_dc77ef61a70c.slice/crio-c8dd8f5e2c57946aa3c9cdcb578095035c03f0107f2e0f0b780a6862eb5dac10 WatchSource:0}: Error finding container c8dd8f5e2c57946aa3c9cdcb578095035c03f0107f2e0f0b780a6862eb5dac10: Status 404 returned error can't find the container with id c8dd8f5e2c57946aa3c9cdcb578095035c03f0107f2e0f0b780a6862eb5dac10
Sep 17 17:41:56.004370 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:56.004347    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/23dbde68-e2d6-4eeb-9132-5f6c29b4d261-console-serving-cert\") pod \"console-5fdd747cc7-gw5hv\" (UID: \"23dbde68-e2d6-4eeb-9132-5f6c29b4d261\") " pod="openshift-console/console-5fdd747cc7-gw5hv"
Sep 17 17:41:56.004467 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:56.004374    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/23dbde68-e2d6-4eeb-9132-5f6c29b4d261-oauth-serving-cert\") pod \"console-5fdd747cc7-gw5hv\" (UID: \"23dbde68-e2d6-4eeb-9132-5f6c29b4d261\") " pod="openshift-console/console-5fdd747cc7-gw5hv"
Sep 17 17:41:56.004467 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:56.004391    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7zcw7\" (UniqueName: \"kubernetes.io/projected/23dbde68-e2d6-4eeb-9132-5f6c29b4d261-kube-api-access-7zcw7\") pod \"console-5fdd747cc7-gw5hv\" (UID: \"23dbde68-e2d6-4eeb-9132-5f6c29b4d261\") " pod="openshift-console/console-5fdd747cc7-gw5hv"
Sep 17 17:41:56.004467 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:56.004411    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/23dbde68-e2d6-4eeb-9132-5f6c29b4d261-console-oauth-config\") pod \"console-5fdd747cc7-gw5hv\" (UID: \"23dbde68-e2d6-4eeb-9132-5f6c29b4d261\") " pod="openshift-console/console-5fdd747cc7-gw5hv"
Sep 17 17:41:56.004467 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:56.004436    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/23dbde68-e2d6-4eeb-9132-5f6c29b4d261-console-config\") pod \"console-5fdd747cc7-gw5hv\" (UID: \"23dbde68-e2d6-4eeb-9132-5f6c29b4d261\") " pod="openshift-console/console-5fdd747cc7-gw5hv"
Sep 17 17:41:56.004654 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:56.004479    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/23dbde68-e2d6-4eeb-9132-5f6c29b4d261-service-ca\") pod \"console-5fdd747cc7-gw5hv\" (UID: \"23dbde68-e2d6-4eeb-9132-5f6c29b4d261\") " pod="openshift-console/console-5fdd747cc7-gw5hv"
Sep 17 17:41:56.005260 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:56.005237    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/23dbde68-e2d6-4eeb-9132-5f6c29b4d261-oauth-serving-cert\") pod \"console-5fdd747cc7-gw5hv\" (UID: \"23dbde68-e2d6-4eeb-9132-5f6c29b4d261\") " pod="openshift-console/console-5fdd747cc7-gw5hv"
Sep 17 17:41:56.005260 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:56.005250    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/23dbde68-e2d6-4eeb-9132-5f6c29b4d261-service-ca\") pod \"console-5fdd747cc7-gw5hv\" (UID: \"23dbde68-e2d6-4eeb-9132-5f6c29b4d261\") " pod="openshift-console/console-5fdd747cc7-gw5hv"
Sep 17 17:41:56.005401 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:56.005248    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/23dbde68-e2d6-4eeb-9132-5f6c29b4d261-console-config\") pod \"console-5fdd747cc7-gw5hv\" (UID: \"23dbde68-e2d6-4eeb-9132-5f6c29b4d261\") " pod="openshift-console/console-5fdd747cc7-gw5hv"
Sep 17 17:41:56.006822 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:56.006804    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/23dbde68-e2d6-4eeb-9132-5f6c29b4d261-console-oauth-config\") pod \"console-5fdd747cc7-gw5hv\" (UID: \"23dbde68-e2d6-4eeb-9132-5f6c29b4d261\") " pod="openshift-console/console-5fdd747cc7-gw5hv"
Sep 17 17:41:56.006916 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:56.006897    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/23dbde68-e2d6-4eeb-9132-5f6c29b4d261-console-serving-cert\") pod \"console-5fdd747cc7-gw5hv\" (UID: \"23dbde68-e2d6-4eeb-9132-5f6c29b4d261\") " pod="openshift-console/console-5fdd747cc7-gw5hv"
Sep 17 17:41:56.019373 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:56.019353    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7zcw7\" (UniqueName: \"kubernetes.io/projected/23dbde68-e2d6-4eeb-9132-5f6c29b4d261-kube-api-access-7zcw7\") pod \"console-5fdd747cc7-gw5hv\" (UID: \"23dbde68-e2d6-4eeb-9132-5f6c29b4d261\") " pod="openshift-console/console-5fdd747cc7-gw5hv"
Sep 17 17:41:56.165453 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:56.165376    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5fdd747cc7-gw5hv"
Sep 17 17:41:56.464654 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:56.464590    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5fdd747cc7-gw5hv"]
Sep 17 17:41:56.467427 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:41:56.467401    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23dbde68_e2d6_4eeb_9132_5f6c29b4d261.slice/crio-2fb0fcd4eb23e2373359cd334d38b011218cf80153f1c4c93b16f577dfe91d65 WatchSource:0}: Error finding container 2fb0fcd4eb23e2373359cd334d38b011218cf80153f1c4c93b16f577dfe91d65: Status 404 returned error can't find the container with id 2fb0fcd4eb23e2373359cd334d38b011218cf80153f1c4c93b16f577dfe91d65
Sep 17 17:41:56.607917 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:56.607863    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5fdd747cc7-gw5hv" event={"ID":"23dbde68-e2d6-4eeb-9132-5f6c29b4d261","Type":"ContainerStarted","Data":"2fb0fcd4eb23e2373359cd334d38b011218cf80153f1c4c93b16f577dfe91d65"}
Sep 17 17:41:56.609195 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:56.609169    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-m66j7" event={"ID":"026b6ea8-2b9e-4f09-97a1-dc77ef61a70c","Type":"ContainerStarted","Data":"c8dd8f5e2c57946aa3c9cdcb578095035c03f0107f2e0f0b780a6862eb5dac10"}
Sep 17 17:41:57.002197 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:56.999747    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:57.070760 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:57.070725    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:57.615256 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:57.615196    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-m66j7" event={"ID":"026b6ea8-2b9e-4f09-97a1-dc77ef61a70c","Type":"ContainerStarted","Data":"9dad89c68891e4d4bf368f7b7af035604f6dd56d0be8cc1c52ba6a874d5ca78b"}
Sep 17 17:41:57.615590 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:57.615561    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-m66j7"
Sep 17 17:41:57.622603 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:57.622578    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-m66j7"
Sep 17 17:41:57.699371 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:57.699290    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-m66j7" podStartSLOduration=2.166960901 podStartE2EDuration="3.699269477s" totalImagesPullingTime="1.532308576s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:41:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:41:57.650266544 +0000 UTC m=+58.673871681" watchObservedRunningTime="2026-09-17 17:41:57.699269477 +0000 UTC m=+58.722874615"
Sep 17 17:41:58.739288 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:58.739248    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:58.778806 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:58.778769    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-7bd5d56d6-v4vgx"]
Sep 17 17:41:58.800197 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:58.800172    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-7bd5d56d6-v4vgx"]
Sep 17 17:41:58.800345 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:58.800277    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-7bd5d56d6-v4vgx"
Sep 17 17:41:58.804690 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:58.804671    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Sep 17 17:41:58.805953 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:58.805931    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-dx4kb\""
Sep 17 17:41:58.806193 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:58.805959    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 17 17:41:58.806193 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:58.805960    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Sep 17 17:41:58.806324 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:58.806209    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 17 17:41:58.806324 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:58.806221    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 17 17:41:58.825850 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:58.825826    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/14c1b35f-ef7b-463f-a407-4205465909ac-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-7bd5d56d6-v4vgx\" (UID: \"14c1b35f-ef7b-463f-a407-4205465909ac\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-v4vgx"
Sep 17 17:41:58.825954 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:58.825860    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/14c1b35f-ef7b-463f-a407-4205465909ac-metrics-client-ca\") pod \"prometheus-operator-7bd5d56d6-v4vgx\" (UID: \"14c1b35f-ef7b-463f-a407-4205465909ac\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-v4vgx"
Sep 17 17:41:58.825954 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:58.825877    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tbb8h\" (UniqueName: \"kubernetes.io/projected/14c1b35f-ef7b-463f-a407-4205465909ac-kube-api-access-tbb8h\") pod \"prometheus-operator-7bd5d56d6-v4vgx\" (UID: \"14c1b35f-ef7b-463f-a407-4205465909ac\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-v4vgx"
Sep 17 17:41:58.825954 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:58.825903    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/14c1b35f-ef7b-463f-a407-4205465909ac-prometheus-operator-tls\") pod \"prometheus-operator-7bd5d56d6-v4vgx\" (UID: \"14c1b35f-ef7b-463f-a407-4205465909ac\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-v4vgx"
Sep 17 17:41:58.840707 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:58.840678    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:58.927104 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:58.927071    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/14c1b35f-ef7b-463f-a407-4205465909ac-metrics-client-ca\") pod \"prometheus-operator-7bd5d56d6-v4vgx\" (UID: \"14c1b35f-ef7b-463f-a407-4205465909ac\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-v4vgx"
Sep 17 17:41:58.927104 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:58.927108    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tbb8h\" (UniqueName: \"kubernetes.io/projected/14c1b35f-ef7b-463f-a407-4205465909ac-kube-api-access-tbb8h\") pod \"prometheus-operator-7bd5d56d6-v4vgx\" (UID: \"14c1b35f-ef7b-463f-a407-4205465909ac\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-v4vgx"
Sep 17 17:41:58.927328 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:58.927148    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/14c1b35f-ef7b-463f-a407-4205465909ac-prometheus-operator-tls\") pod \"prometheus-operator-7bd5d56d6-v4vgx\" (UID: \"14c1b35f-ef7b-463f-a407-4205465909ac\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-v4vgx"
Sep 17 17:41:58.927328 ip-10-0-139-237 kubenswrapper[2567]: E0917 17:41:58.927280    2567 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found
Sep 17 17:41:58.927436 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:58.927318    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/14c1b35f-ef7b-463f-a407-4205465909ac-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-7bd5d56d6-v4vgx\" (UID: \"14c1b35f-ef7b-463f-a407-4205465909ac\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-v4vgx"
Sep 17 17:41:58.927436 ip-10-0-139-237 kubenswrapper[2567]: E0917 17:41:58.927336    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/14c1b35f-ef7b-463f-a407-4205465909ac-prometheus-operator-tls podName:14c1b35f-ef7b-463f-a407-4205465909ac nodeName:}" failed. No retries permitted until 2026-09-17 17:41:59.427318394 +0000 UTC m=+60.450923524 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/14c1b35f-ef7b-463f-a407-4205465909ac-prometheus-operator-tls") pod "prometheus-operator-7bd5d56d6-v4vgx" (UID: "14c1b35f-ef7b-463f-a407-4205465909ac") : secret "prometheus-operator-tls" not found
Sep 17 17:41:58.927838 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:58.927817    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/14c1b35f-ef7b-463f-a407-4205465909ac-metrics-client-ca\") pod \"prometheus-operator-7bd5d56d6-v4vgx\" (UID: \"14c1b35f-ef7b-463f-a407-4205465909ac\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-v4vgx"
Sep 17 17:41:58.929705 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:58.929684    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/14c1b35f-ef7b-463f-a407-4205465909ac-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-7bd5d56d6-v4vgx\" (UID: \"14c1b35f-ef7b-463f-a407-4205465909ac\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-v4vgx"
Sep 17 17:41:58.936080 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:58.936054    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tbb8h\" (UniqueName: \"kubernetes.io/projected/14c1b35f-ef7b-463f-a407-4205465909ac-kube-api-access-tbb8h\") pod \"prometheus-operator-7bd5d56d6-v4vgx\" (UID: \"14c1b35f-ef7b-463f-a407-4205465909ac\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-v4vgx"
Sep 17 17:41:59.113150 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:59.113116    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:59.431089 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:59.431004    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/14c1b35f-ef7b-463f-a407-4205465909ac-prometheus-operator-tls\") pod \"prometheus-operator-7bd5d56d6-v4vgx\" (UID: \"14c1b35f-ef7b-463f-a407-4205465909ac\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-v4vgx"
Sep 17 17:41:59.433610 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:59.433585    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/14c1b35f-ef7b-463f-a407-4205465909ac-prometheus-operator-tls\") pod \"prometheus-operator-7bd5d56d6-v4vgx\" (UID: \"14c1b35f-ef7b-463f-a407-4205465909ac\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-v4vgx"
Sep 17 17:41:59.708654 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:41:59.708589    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-7bd5d56d6-v4vgx"
Sep 17 17:42:00.533266 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:00.533239    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-vwtvz"
Sep 17 17:42:00.798398 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:00.798360    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 17:42:00.904831 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:00.904780    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 17:42:01.044117 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:01.044091    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-7bd5d56d6-v4vgx"]
Sep 17 17:42:01.047099 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:42:01.047077    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14c1b35f_ef7b_463f_a407_4205465909ac.slice/crio-a74c38a4eff2c34f4eba5c69e979d3a825d7d03051313f43d938a26841342a97 WatchSource:0}: Error finding container a74c38a4eff2c34f4eba5c69e979d3a825d7d03051313f43d938a26841342a97: Status 404 returned error can't find the container with id a74c38a4eff2c34f4eba5c69e979d3a825d7d03051313f43d938a26841342a97
Sep 17 17:42:01.399160 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:01.399073    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 17:42:01.465587 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:01.465558    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 17:42:01.630985 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:01.630951    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5fdd747cc7-gw5hv" event={"ID":"23dbde68-e2d6-4eeb-9132-5f6c29b4d261","Type":"ContainerStarted","Data":"f3220517bcf3d3ddbe319912a49a1602a3c5de5dfd98df22b770e04aa5c8f4a4"}
Sep 17 17:42:01.632386 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:01.632326    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-7bd5d56d6-v4vgx" event={"ID":"14c1b35f-ef7b-463f-a407-4205465909ac","Type":"ContainerStarted","Data":"a74c38a4eff2c34f4eba5c69e979d3a825d7d03051313f43d938a26841342a97"}
Sep 17 17:42:01.654280 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:01.654195    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5fdd747cc7-gw5hv" podStartSLOduration=2.308118903 podStartE2EDuration="6.654176342s" totalImagesPullingTime="4.346057439s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:41:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:42:01.652713376 +0000 UTC m=+62.676318536" watchObservedRunningTime="2026-09-17 17:42:01.654176342 +0000 UTC m=+62.677781480"
Sep 17 17:42:02.467236 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:02.467148    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 17:42:02.641334 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:02.641296    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-7bd5d56d6-v4vgx" event={"ID":"14c1b35f-ef7b-463f-a407-4205465909ac","Type":"ContainerStarted","Data":"ab81fc4bee4b154530389c90ded66e995c57fba99f064b7612235afdc5694c6c"}
Sep 17 17:42:02.641804 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:02.641387    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-7bd5d56d6-v4vgx" event={"ID":"14c1b35f-ef7b-463f-a407-4205465909ac","Type":"ContainerStarted","Data":"7ef02cf03f72863a7611d900906a52b612ef12f92caf657fb89954f1ee303e1f"}
Sep 17 17:42:02.660619 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:02.660574    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-7bd5d56d6-v4vgx" podStartSLOduration=3.527702492 podStartE2EDuration="4.660561488s" totalImagesPullingTime="1.132858996s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:41:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:42:02.660177584 +0000 UTC m=+63.683782739" watchObservedRunningTime="2026-09-17 17:42:02.660561488 +0000 UTC m=+63.684166624"
Sep 17 17:42:03.602047 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:03.602019    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-mdbmq"
Sep 17 17:42:03.924641 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:03.924561    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 17:42:03.994836 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:03.994809    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-74b7bc945c-5f8mh"]
Sep 17 17:42:03.996896 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:03.996876    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-74b7bc945c-5f8mh"
Sep 17 17:42:04.007706 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:04.007678    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Sep 17 17:42:04.012144 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:04.012124    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-74b7bc945c-5f8mh"]
Sep 17 17:42:04.016020 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:04.015997    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 17:42:04.066694 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:04.066667    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/338c12c5-2bd0-444a-8fb6-276caf969978-console-config\") pod \"console-74b7bc945c-5f8mh\" (UID: \"338c12c5-2bd0-444a-8fb6-276caf969978\") " pod="openshift-console/console-74b7bc945c-5f8mh"
Sep 17 17:42:04.066791 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:04.066704    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/338c12c5-2bd0-444a-8fb6-276caf969978-service-ca\") pod \"console-74b7bc945c-5f8mh\" (UID: \"338c12c5-2bd0-444a-8fb6-276caf969978\") " pod="openshift-console/console-74b7bc945c-5f8mh"
Sep 17 17:42:04.066791 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:04.066749    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/338c12c5-2bd0-444a-8fb6-276caf969978-console-serving-cert\") pod \"console-74b7bc945c-5f8mh\" (UID: \"338c12c5-2bd0-444a-8fb6-276caf969978\") " pod="openshift-console/console-74b7bc945c-5f8mh"
Sep 17 17:42:04.066874 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:04.066791    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/338c12c5-2bd0-444a-8fb6-276caf969978-console-oauth-config\") pod \"console-74b7bc945c-5f8mh\" (UID: \"338c12c5-2bd0-444a-8fb6-276caf969978\") " pod="openshift-console/console-74b7bc945c-5f8mh"
Sep 17 17:42:04.066874 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:04.066819    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gjdqf\" (UniqueName: \"kubernetes.io/projected/338c12c5-2bd0-444a-8fb6-276caf969978-kube-api-access-gjdqf\") pod \"console-74b7bc945c-5f8mh\" (UID: \"338c12c5-2bd0-444a-8fb6-276caf969978\") " pod="openshift-console/console-74b7bc945c-5f8mh"
Sep 17 17:42:04.066874 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:04.066839    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/338c12c5-2bd0-444a-8fb6-276caf969978-trusted-ca-bundle\") pod \"console-74b7bc945c-5f8mh\" (UID: \"338c12c5-2bd0-444a-8fb6-276caf969978\") " pod="openshift-console/console-74b7bc945c-5f8mh"
Sep 17 17:42:04.066965 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:04.066882    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/338c12c5-2bd0-444a-8fb6-276caf969978-oauth-serving-cert\") pod \"console-74b7bc945c-5f8mh\" (UID: \"338c12c5-2bd0-444a-8fb6-276caf969978\") " pod="openshift-console/console-74b7bc945c-5f8mh"
Sep 17 17:42:04.143154 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:04.143121    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 17:42:04.167679 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:04.167655    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/338c12c5-2bd0-444a-8fb6-276caf969978-console-serving-cert\") pod \"console-74b7bc945c-5f8mh\" (UID: \"338c12c5-2bd0-444a-8fb6-276caf969978\") " pod="openshift-console/console-74b7bc945c-5f8mh"
Sep 17 17:42:04.167786 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:04.167683    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/338c12c5-2bd0-444a-8fb6-276caf969978-console-oauth-config\") pod \"console-74b7bc945c-5f8mh\" (UID: \"338c12c5-2bd0-444a-8fb6-276caf969978\") " pod="openshift-console/console-74b7bc945c-5f8mh"
Sep 17 17:42:04.167786 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:04.167701    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gjdqf\" (UniqueName: \"kubernetes.io/projected/338c12c5-2bd0-444a-8fb6-276caf969978-kube-api-access-gjdqf\") pod \"console-74b7bc945c-5f8mh\" (UID: \"338c12c5-2bd0-444a-8fb6-276caf969978\") " pod="openshift-console/console-74b7bc945c-5f8mh"
Sep 17 17:42:04.167786 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:04.167719    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/338c12c5-2bd0-444a-8fb6-276caf969978-trusted-ca-bundle\") pod \"console-74b7bc945c-5f8mh\" (UID: \"338c12c5-2bd0-444a-8fb6-276caf969978\") " pod="openshift-console/console-74b7bc945c-5f8mh"
Sep 17 17:42:04.167786 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:04.167766    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/338c12c5-2bd0-444a-8fb6-276caf969978-oauth-serving-cert\") pod \"console-74b7bc945c-5f8mh\" (UID: \"338c12c5-2bd0-444a-8fb6-276caf969978\") " pod="openshift-console/console-74b7bc945c-5f8mh"
Sep 17 17:42:04.167997 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:04.167906    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/338c12c5-2bd0-444a-8fb6-276caf969978-console-config\") pod \"console-74b7bc945c-5f8mh\" (UID: \"338c12c5-2bd0-444a-8fb6-276caf969978\") " pod="openshift-console/console-74b7bc945c-5f8mh"
Sep 17 17:42:04.167997 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:04.167958    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/338c12c5-2bd0-444a-8fb6-276caf969978-service-ca\") pod \"console-74b7bc945c-5f8mh\" (UID: \"338c12c5-2bd0-444a-8fb6-276caf969978\") " pod="openshift-console/console-74b7bc945c-5f8mh"
Sep 17 17:42:04.168561 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:04.168520    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/338c12c5-2bd0-444a-8fb6-276caf969978-service-ca\") pod \"console-74b7bc945c-5f8mh\" (UID: \"338c12c5-2bd0-444a-8fb6-276caf969978\") " pod="openshift-console/console-74b7bc945c-5f8mh"
Sep 17 17:42:04.168645 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:04.168581    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/338c12c5-2bd0-444a-8fb6-276caf969978-oauth-serving-cert\") pod \"console-74b7bc945c-5f8mh\" (UID: \"338c12c5-2bd0-444a-8fb6-276caf969978\") " pod="openshift-console/console-74b7bc945c-5f8mh"
Sep 17 17:42:04.168645 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:04.168634    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/338c12c5-2bd0-444a-8fb6-276caf969978-trusted-ca-bundle\") pod \"console-74b7bc945c-5f8mh\" (UID: \"338c12c5-2bd0-444a-8fb6-276caf969978\") " pod="openshift-console/console-74b7bc945c-5f8mh"
Sep 17 17:42:04.168728 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:04.168654    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/338c12c5-2bd0-444a-8fb6-276caf969978-console-config\") pod \"console-74b7bc945c-5f8mh\" (UID: \"338c12c5-2bd0-444a-8fb6-276caf969978\") " pod="openshift-console/console-74b7bc945c-5f8mh"
Sep 17 17:42:04.170638 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:04.170621    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/338c12c5-2bd0-444a-8fb6-276caf969978-console-serving-cert\") pod \"console-74b7bc945c-5f8mh\" (UID: \"338c12c5-2bd0-444a-8fb6-276caf969978\") " pod="openshift-console/console-74b7bc945c-5f8mh"
Sep 17 17:42:04.170733 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:04.170688    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/338c12c5-2bd0-444a-8fb6-276caf969978-console-oauth-config\") pod \"console-74b7bc945c-5f8mh\" (UID: \"338c12c5-2bd0-444a-8fb6-276caf969978\") " pod="openshift-console/console-74b7bc945c-5f8mh"
Sep 17 17:42:04.185365 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:04.185316    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-84c885658c-wg927"]
Sep 17 17:42:04.188081 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:04.188068    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-84c885658c-wg927"
Sep 17 17:42:04.192997 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:04.192980    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Sep 17 17:42:04.193095 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:04.193081    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Sep 17 17:42:04.193148 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:04.193089    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Sep 17 17:42:04.193369 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:04.193352    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-9k9f2\""
Sep 17 17:42:04.195653 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:04.195637    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gjdqf\" (UniqueName: \"kubernetes.io/projected/338c12c5-2bd0-444a-8fb6-276caf969978-kube-api-access-gjdqf\") pod \"console-74b7bc945c-5f8mh\" (UID: \"338c12c5-2bd0-444a-8fb6-276caf969978\") " pod="openshift-console/console-74b7bc945c-5f8mh"
Sep 17 17:42:04.210522 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:04.210500    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-wdlzx"]
Sep 17 17:42:04.212709 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:04.212693    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-wdlzx"
Sep 17 17:42:04.214800 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:04.214777    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-84c885658c-wg927"]
Sep 17 17:42:04.225908 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:04.225890    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 17 17:42:04.226034 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:04.225921    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 17 17:42:04.226034 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:04.226017    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-x5mb7\""
Sep 17 17:42:04.226220 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:04.226207    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 17 17:42:04.269110 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:04.269081    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/92415019-ec6d-46b5-99b0-0be3386db8bc-metrics-client-ca\") pod \"node-exporter-wdlzx\" (UID: \"92415019-ec6d-46b5-99b0-0be3386db8bc\") " pod="openshift-monitoring/node-exporter-wdlzx"
Sep 17 17:42:04.269233 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:04.269122    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/92415019-ec6d-46b5-99b0-0be3386db8bc-root\") pod \"node-exporter-wdlzx\" (UID: \"92415019-ec6d-46b5-99b0-0be3386db8bc\") " pod="openshift-monitoring/node-exporter-wdlzx"
Sep 17 17:42:04.269233 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:04.269145    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/92415019-ec6d-46b5-99b0-0be3386db8bc-node-exporter-wtmp\") pod \"node-exporter-wdlzx\" (UID: \"92415019-ec6d-46b5-99b0-0be3386db8bc\") " pod="openshift-monitoring/node-exporter-wdlzx"
Sep 17 17:42:04.269233 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:04.269208    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/db8f376a-5499-4162-b13f-f552de771aae-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-84c885658c-wg927\" (UID: \"db8f376a-5499-4162-b13f-f552de771aae\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-wg927"
Sep 17 17:42:04.269366 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:04.269238    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/db8f376a-5499-4162-b13f-f552de771aae-metrics-client-ca\") pod \"kube-state-metrics-84c885658c-wg927\" (UID: \"db8f376a-5499-4162-b13f-f552de771aae\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-wg927"
Sep 17 17:42:04.269366 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:04.269268    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/92415019-ec6d-46b5-99b0-0be3386db8bc-sys\") pod \"node-exporter-wdlzx\" (UID: \"92415019-ec6d-46b5-99b0-0be3386db8bc\") " pod="openshift-monitoring/node-exporter-wdlzx"
Sep 17 17:42:04.269366 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:04.269303    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/92415019-ec6d-46b5-99b0-0be3386db8bc-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-wdlzx\" (UID: \"92415019-ec6d-46b5-99b0-0be3386db8bc\") " pod="openshift-monitoring/node-exporter-wdlzx"
Sep 17 17:42:04.269366 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:04.269327    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/db8f376a-5499-4162-b13f-f552de771aae-volume-directive-shadow\") pod \"kube-state-metrics-84c885658c-wg927\" (UID: \"db8f376a-5499-4162-b13f-f552de771aae\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-wg927"
Sep 17 17:42:04.269519 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:04.269378    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/92415019-ec6d-46b5-99b0-0be3386db8bc-node-exporter-textfile\") pod \"node-exporter-wdlzx\" (UID: \"92415019-ec6d-46b5-99b0-0be3386db8bc\") " pod="openshift-monitoring/node-exporter-wdlzx"
Sep 17 17:42:04.269519 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:04.269413    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/db8f376a-5499-4162-b13f-f552de771aae-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-84c885658c-wg927\" (UID: \"db8f376a-5499-4162-b13f-f552de771aae\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-wg927"
Sep 17 17:42:04.269519 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:04.269444    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2nr7z\" (UniqueName: \"kubernetes.io/projected/92415019-ec6d-46b5-99b0-0be3386db8bc-kube-api-access-2nr7z\") pod \"node-exporter-wdlzx\" (UID: \"92415019-ec6d-46b5-99b0-0be3386db8bc\") " pod="openshift-monitoring/node-exporter-wdlzx"
Sep 17 17:42:04.269519 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:04.269465    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/92415019-ec6d-46b5-99b0-0be3386db8bc-node-exporter-accelerators-collector-config\") pod \"node-exporter-wdlzx\" (UID: \"92415019-ec6d-46b5-99b0-0be3386db8bc\") " pod="openshift-monitoring/node-exporter-wdlzx"
Sep 17 17:42:04.269519 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:04.269479    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/db8f376a-5499-4162-b13f-f552de771aae-kube-state-metrics-tls\") pod \"kube-state-metrics-84c885658c-wg927\" (UID: \"db8f376a-5499-4162-b13f-f552de771aae\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-wg927"
Sep 17 17:42:04.269519 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:04.269509    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jcxns\" (UniqueName: \"kubernetes.io/projected/db8f376a-5499-4162-b13f-f552de771aae-kube-api-access-jcxns\") pod \"kube-state-metrics-84c885658c-wg927\" (UID: \"db8f376a-5499-4162-b13f-f552de771aae\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-wg927"
Sep 17 17:42:04.269743 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:04.269526    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/92415019-ec6d-46b5-99b0-0be3386db8bc-node-exporter-tls\") pod \"node-exporter-wdlzx\" (UID: \"92415019-ec6d-46b5-99b0-0be3386db8bc\") " pod="openshift-monitoring/node-exporter-wdlzx"
Sep 17 17:42:04.284919 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:04.284892    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 17:42:04.305854 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:04.305829    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-74b7bc945c-5f8mh"
Sep 17 17:42:04.370631 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:04.370600    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/92415019-ec6d-46b5-99b0-0be3386db8bc-metrics-client-ca\") pod \"node-exporter-wdlzx\" (UID: \"92415019-ec6d-46b5-99b0-0be3386db8bc\") " pod="openshift-monitoring/node-exporter-wdlzx"
Sep 17 17:42:04.370770 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:04.370635    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/92415019-ec6d-46b5-99b0-0be3386db8bc-root\") pod \"node-exporter-wdlzx\" (UID: \"92415019-ec6d-46b5-99b0-0be3386db8bc\") " pod="openshift-monitoring/node-exporter-wdlzx"
Sep 17 17:42:04.370770 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:04.370652    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/92415019-ec6d-46b5-99b0-0be3386db8bc-node-exporter-wtmp\") pod \"node-exporter-wdlzx\" (UID: \"92415019-ec6d-46b5-99b0-0be3386db8bc\") " pod="openshift-monitoring/node-exporter-wdlzx"
Sep 17 17:42:04.370770 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:04.370700    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/92415019-ec6d-46b5-99b0-0be3386db8bc-root\") pod \"node-exporter-wdlzx\" (UID: \"92415019-ec6d-46b5-99b0-0be3386db8bc\") " pod="openshift-monitoring/node-exporter-wdlzx"
Sep 17 17:42:04.370770 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:04.370728    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/db8f376a-5499-4162-b13f-f552de771aae-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-84c885658c-wg927\" (UID: \"db8f376a-5499-4162-b13f-f552de771aae\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-wg927"
Sep 17 17:42:04.370770 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:04.370747    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/db8f376a-5499-4162-b13f-f552de771aae-metrics-client-ca\") pod \"kube-state-metrics-84c885658c-wg927\" (UID: \"db8f376a-5499-4162-b13f-f552de771aae\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-wg927"
Sep 17 17:42:04.371021 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:04.370771    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/92415019-ec6d-46b5-99b0-0be3386db8bc-sys\") pod \"node-exporter-wdlzx\" (UID: \"92415019-ec6d-46b5-99b0-0be3386db8bc\") " pod="openshift-monitoring/node-exporter-wdlzx"
Sep 17 17:42:04.371021 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:04.370835    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/92415019-ec6d-46b5-99b0-0be3386db8bc-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-wdlzx\" (UID: \"92415019-ec6d-46b5-99b0-0be3386db8bc\") " pod="openshift-monitoring/node-exporter-wdlzx"
Sep 17 17:42:04.371021 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:04.370937    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/92415019-ec6d-46b5-99b0-0be3386db8bc-node-exporter-wtmp\") pod \"node-exporter-wdlzx\" (UID: \"92415019-ec6d-46b5-99b0-0be3386db8bc\") " pod="openshift-monitoring/node-exporter-wdlzx"
Sep 17 17:42:04.371021 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:04.371006    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/92415019-ec6d-46b5-99b0-0be3386db8bc-sys\") pod \"node-exporter-wdlzx\" (UID: \"92415019-ec6d-46b5-99b0-0be3386db8bc\") " pod="openshift-monitoring/node-exporter-wdlzx"
Sep 17 17:42:04.371222 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:04.371112    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/db8f376a-5499-4162-b13f-f552de771aae-volume-directive-shadow\") pod \"kube-state-metrics-84c885658c-wg927\" (UID: \"db8f376a-5499-4162-b13f-f552de771aae\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-wg927"
Sep 17 17:42:04.371222 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:04.371168    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/92415019-ec6d-46b5-99b0-0be3386db8bc-node-exporter-textfile\") pod \"node-exporter-wdlzx\" (UID: \"92415019-ec6d-46b5-99b0-0be3386db8bc\") " pod="openshift-monitoring/node-exporter-wdlzx"
Sep 17 17:42:04.371222 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:04.371205    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/db8f376a-5499-4162-b13f-f552de771aae-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-84c885658c-wg927\" (UID: \"db8f376a-5499-4162-b13f-f552de771aae\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-wg927"
Sep 17 17:42:04.371369 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:04.371253    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2nr7z\" (UniqueName: \"kubernetes.io/projected/92415019-ec6d-46b5-99b0-0be3386db8bc-kube-api-access-2nr7z\") pod \"node-exporter-wdlzx\" (UID: \"92415019-ec6d-46b5-99b0-0be3386db8bc\") " pod="openshift-monitoring/node-exporter-wdlzx"
Sep 17 17:42:04.371369 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:04.371285    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/92415019-ec6d-46b5-99b0-0be3386db8bc-node-exporter-accelerators-collector-config\") pod \"node-exporter-wdlzx\" (UID: \"92415019-ec6d-46b5-99b0-0be3386db8bc\") " pod="openshift-monitoring/node-exporter-wdlzx"
Sep 17 17:42:04.371369 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:04.371312    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/db8f376a-5499-4162-b13f-f552de771aae-kube-state-metrics-tls\") pod \"kube-state-metrics-84c885658c-wg927\" (UID: \"db8f376a-5499-4162-b13f-f552de771aae\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-wg927"
Sep 17 17:42:04.371369 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:04.371348    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jcxns\" (UniqueName: \"kubernetes.io/projected/db8f376a-5499-4162-b13f-f552de771aae-kube-api-access-jcxns\") pod \"kube-state-metrics-84c885658c-wg927\" (UID: \"db8f376a-5499-4162-b13f-f552de771aae\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-wg927"
Sep 17 17:42:04.371589 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:04.371376    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/92415019-ec6d-46b5-99b0-0be3386db8bc-node-exporter-tls\") pod \"node-exporter-wdlzx\" (UID: \"92415019-ec6d-46b5-99b0-0be3386db8bc\") " pod="openshift-monitoring/node-exporter-wdlzx"
Sep 17 17:42:04.371589 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:04.371489    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/db8f376a-5499-4162-b13f-f552de771aae-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-84c885658c-wg927\" (UID: \"db8f376a-5499-4162-b13f-f552de771aae\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-wg927"
Sep 17 17:42:04.371589 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:04.371312    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/92415019-ec6d-46b5-99b0-0be3386db8bc-metrics-client-ca\") pod \"node-exporter-wdlzx\" (UID: \"92415019-ec6d-46b5-99b0-0be3386db8bc\") " pod="openshift-monitoring/node-exporter-wdlzx"
Sep 17 17:42:04.371741 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:04.371609    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/92415019-ec6d-46b5-99b0-0be3386db8bc-node-exporter-textfile\") pod \"node-exporter-wdlzx\" (UID: \"92415019-ec6d-46b5-99b0-0be3386db8bc\") " pod="openshift-monitoring/node-exporter-wdlzx"
Sep 17 17:42:04.371741 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:04.371618    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/db8f376a-5499-4162-b13f-f552de771aae-metrics-client-ca\") pod \"kube-state-metrics-84c885658c-wg927\" (UID: \"db8f376a-5499-4162-b13f-f552de771aae\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-wg927"
Sep 17 17:42:04.371837 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:04.371805    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/92415019-ec6d-46b5-99b0-0be3386db8bc-node-exporter-accelerators-collector-config\") pod \"node-exporter-wdlzx\" (UID: \"92415019-ec6d-46b5-99b0-0be3386db8bc\") " pod="openshift-monitoring/node-exporter-wdlzx"
Sep 17 17:42:04.371999 ip-10-0-139-237 kubenswrapper[2567]: E0917 17:42:04.371981    2567 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found
Sep 17 17:42:04.372059 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:04.372024    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/db8f376a-5499-4162-b13f-f552de771aae-volume-directive-shadow\") pod \"kube-state-metrics-84c885658c-wg927\" (UID: \"db8f376a-5499-4162-b13f-f552de771aae\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-wg927"
Sep 17 17:42:04.372059 ip-10-0-139-237 kubenswrapper[2567]: E0917 17:42:04.372054    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/db8f376a-5499-4162-b13f-f552de771aae-kube-state-metrics-tls podName:db8f376a-5499-4162-b13f-f552de771aae nodeName:}" failed. No retries permitted until 2026-09-17 17:42:04.872034133 +0000 UTC m=+65.895639257 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/db8f376a-5499-4162-b13f-f552de771aae-kube-state-metrics-tls") pod "kube-state-metrics-84c885658c-wg927" (UID: "db8f376a-5499-4162-b13f-f552de771aae") : secret "kube-state-metrics-tls" not found
Sep 17 17:42:04.373956 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:04.373934    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/92415019-ec6d-46b5-99b0-0be3386db8bc-node-exporter-tls\") pod \"node-exporter-wdlzx\" (UID: \"92415019-ec6d-46b5-99b0-0be3386db8bc\") " pod="openshift-monitoring/node-exporter-wdlzx"
Sep 17 17:42:04.374046 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:04.374006    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/db8f376a-5499-4162-b13f-f552de771aae-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-84c885658c-wg927\" (UID: \"db8f376a-5499-4162-b13f-f552de771aae\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-wg927"
Sep 17 17:42:04.374046 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:04.374006    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/92415019-ec6d-46b5-99b0-0be3386db8bc-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-wdlzx\" (UID: \"92415019-ec6d-46b5-99b0-0be3386db8bc\") " pod="openshift-monitoring/node-exporter-wdlzx"
Sep 17 17:42:04.390687 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:04.390668    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jcxns\" (UniqueName: \"kubernetes.io/projected/db8f376a-5499-4162-b13f-f552de771aae-kube-api-access-jcxns\") pod \"kube-state-metrics-84c885658c-wg927\" (UID: \"db8f376a-5499-4162-b13f-f552de771aae\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-wg927"
Sep 17 17:42:04.395210 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:04.395189    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2nr7z\" (UniqueName: \"kubernetes.io/projected/92415019-ec6d-46b5-99b0-0be3386db8bc-kube-api-access-2nr7z\") pod \"node-exporter-wdlzx\" (UID: \"92415019-ec6d-46b5-99b0-0be3386db8bc\") " pod="openshift-monitoring/node-exporter-wdlzx"
Sep 17 17:42:04.433382 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:04.433347    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 17:42:04.521269 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:04.521207    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-wdlzx"
Sep 17 17:42:04.530920 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:42:04.530892    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92415019_ec6d_46b5_99b0_0be3386db8bc.slice/crio-71e3ce9a5536d0836cbcd8d46c3d9e942ad422e0a0d521dd18ac897fe66758af WatchSource:0}: Error finding container 71e3ce9a5536d0836cbcd8d46c3d9e942ad422e0a0d521dd18ac897fe66758af: Status 404 returned error can't find the container with id 71e3ce9a5536d0836cbcd8d46c3d9e942ad422e0a0d521dd18ac897fe66758af
Sep 17 17:42:04.564732 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:04.564698    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 17:42:04.581408 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:04.581385    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-74b7bc945c-5f8mh"]
Sep 17 17:42:04.584450 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:42:04.584428    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod338c12c5_2bd0_444a_8fb6_276caf969978.slice/crio-ae558472090125cb4df039be0089ac4f4e9e6b50682944346febc72f41fb9014 WatchSource:0}: Error finding container ae558472090125cb4df039be0089ac4f4e9e6b50682944346febc72f41fb9014: Status 404 returned error can't find the container with id ae558472090125cb4df039be0089ac4f4e9e6b50682944346febc72f41fb9014
Sep 17 17:42:04.650017 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:04.649984    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-74b7bc945c-5f8mh" event={"ID":"338c12c5-2bd0-444a-8fb6-276caf969978","Type":"ContainerStarted","Data":"efa07b0506a0817e95a3189b4ff0ff799d09f64e8cef19ffc9ee9cad3aa234c3"}
Sep 17 17:42:04.650144 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:04.650048    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-74b7bc945c-5f8mh" event={"ID":"338c12c5-2bd0-444a-8fb6-276caf969978","Type":"ContainerStarted","Data":"ae558472090125cb4df039be0089ac4f4e9e6b50682944346febc72f41fb9014"}
Sep 17 17:42:04.651202 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:04.651174    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-wdlzx" event={"ID":"92415019-ec6d-46b5-99b0-0be3386db8bc","Type":"ContainerStarted","Data":"71e3ce9a5536d0836cbcd8d46c3d9e942ad422e0a0d521dd18ac897fe66758af"}
Sep 17 17:42:04.758713 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:04.758519    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 17:42:04.876977 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:04.876944    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/db8f376a-5499-4162-b13f-f552de771aae-kube-state-metrics-tls\") pod \"kube-state-metrics-84c885658c-wg927\" (UID: \"db8f376a-5499-4162-b13f-f552de771aae\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-wg927"
Sep 17 17:42:04.879249 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:04.879232    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/db8f376a-5499-4162-b13f-f552de771aae-kube-state-metrics-tls\") pod \"kube-state-metrics-84c885658c-wg927\" (UID: \"db8f376a-5499-4162-b13f-f552de771aae\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-wg927"
Sep 17 17:42:04.880771 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:04.880728    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-74b7bc945c-5f8mh" podStartSLOduration=1.880715088 podStartE2EDuration="1.880715088s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:42:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:42:04.870697949 +0000 UTC m=+65.894303085" watchObservedRunningTime="2026-09-17 17:42:04.880715088 +0000 UTC m=+65.904320223"
Sep 17 17:42:05.017911 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:05.017876    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 17:42:05.097505 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:05.097475    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-84c885658c-wg927"
Sep 17 17:42:05.135333 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:05.135253    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 17:42:05.248146 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:05.248100    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 17:42:05.354393 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:05.354347    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 17:42:05.395691 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:05.395607    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-84c885658c-wg927"]
Sep 17 17:42:05.427595 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:42:05.427561    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb8f376a_5499_4162_b13f_f552de771aae.slice/crio-2e5ff4a61ef7d777b5cac18697bac0a30c53c8f2c5212d1d47349519357ff8ac WatchSource:0}: Error finding container 2e5ff4a61ef7d777b5cac18697bac0a30c53c8f2c5212d1d47349519357ff8ac: Status 404 returned error can't find the container with id 2e5ff4a61ef7d777b5cac18697bac0a30c53c8f2c5212d1d47349519357ff8ac
Sep 17 17:42:05.487808 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:05.487780    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 17:42:05.580889 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:05.580853    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 17:42:05.656099 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:05.656038    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-84c885658c-wg927" event={"ID":"db8f376a-5499-4162-b13f-f552de771aae","Type":"ContainerStarted","Data":"2e5ff4a61ef7d777b5cac18697bac0a30c53c8f2c5212d1d47349519357ff8ac"}
Sep 17 17:42:05.657607 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:05.657584    2567 generic.go:356] "Generic (PLEG): container finished" podID="92415019-ec6d-46b5-99b0-0be3386db8bc" containerID="23d765d2b4138d723b7689f80f74423196e68f38468af062f3415db6869f33e1" exitCode=0
Sep 17 17:42:05.657709 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:05.657685    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-wdlzx" event={"ID":"92415019-ec6d-46b5-99b0-0be3386db8bc","Type":"ContainerDied","Data":"23d765d2b4138d723b7689f80f74423196e68f38468af062f3415db6869f33e1"}
Sep 17 17:42:05.778110 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:05.778083    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 17:42:06.165758 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:06.165715    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5fdd747cc7-gw5hv"
Sep 17 17:42:06.166150 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:06.165788    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5fdd747cc7-gw5hv"
Sep 17 17:42:06.170764 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:06.170742    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5fdd747cc7-gw5hv"
Sep 17 17:42:06.662602 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:06.662568    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-wdlzx" event={"ID":"92415019-ec6d-46b5-99b0-0be3386db8bc","Type":"ContainerStarted","Data":"7acf7e8c907b0eac771d629f7fbea30ada29440391419b931edac8b06c75b652"}
Sep 17 17:42:06.662833 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:06.662620    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-wdlzx" event={"ID":"92415019-ec6d-46b5-99b0-0be3386db8bc","Type":"ContainerStarted","Data":"73b217f92c53d74ac62dbe2f129e5f39b8b74d031e735ad22b2fabc146b05b90"}
Sep 17 17:42:06.666358 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:06.666337    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5fdd747cc7-gw5hv"
Sep 17 17:42:06.685642 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:06.685600    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-wdlzx" podStartSLOduration=1.699925684 podStartE2EDuration="2.685585959s" totalImagesPullingTime="919.527631ms" totalInitContainerRuntime="66.132644ms" isStatefulPod=true podCreationTimestamp="2026-09-17 17:42:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:42:06.683830912 +0000 UTC m=+67.707436049" watchObservedRunningTime="2026-09-17 17:42:06.685585959 +0000 UTC m=+67.709191094"
Sep 17 17:42:06.997273 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:06.997247    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/db73a8fe-d34c-41c8-8ba4-8040b4bacf13-original-pull-secret\") pod \"global-pull-secret-syncer-vjstm\" (UID: \"db73a8fe-d34c-41c8-8ba4-8040b4bacf13\") " pod="kube-system/global-pull-secret-syncer-vjstm"
Sep 17 17:42:06.997373 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:06.997281    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5ab7aae3-053b-4e36-9c4a-c523f0cfb615-metrics-certs\") pod \"network-metrics-daemon-xg2wr\" (UID: \"5ab7aae3-053b-4e36-9c4a-c523f0cfb615\") " pod="openshift-multus/network-metrics-daemon-xg2wr"
Sep 17 17:42:07.001666 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:07.001649    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 17 17:42:07.001915 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:07.001897    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 17 17:42:07.009631 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:07.009612    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5ab7aae3-053b-4e36-9c4a-c523f0cfb615-metrics-certs\") pod \"network-metrics-daemon-xg2wr\" (UID: \"5ab7aae3-053b-4e36-9c4a-c523f0cfb615\") " pod="openshift-multus/network-metrics-daemon-xg2wr"
Sep 17 17:42:07.009714 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:07.009635    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/db73a8fe-d34c-41c8-8ba4-8040b4bacf13-original-pull-secret\") pod \"global-pull-secret-syncer-vjstm\" (UID: \"db73a8fe-d34c-41c8-8ba4-8040b4bacf13\") " pod="kube-system/global-pull-secret-syncer-vjstm"
Sep 17 17:42:07.094934 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:07.094911    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-jc79n\""
Sep 17 17:42:07.098524 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:07.098506    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vjstm"
Sep 17 17:42:07.101256 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:07.101237    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xg2wr"
Sep 17 17:42:07.201317 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:07.199006    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lndx7\" (UniqueName: \"kubernetes.io/projected/6d0a7973-cc86-4fda-9056-11b5923bd14c-kube-api-access-lndx7\") pod \"network-check-target-4cb4f\" (UID: \"6d0a7973-cc86-4fda-9056-11b5923bd14c\") " pod="openshift-network-diagnostics/network-check-target-4cb4f"
Sep 17 17:42:07.207197 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:07.207179    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 17 17:42:07.217057 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:07.216811    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 17 17:42:07.229191 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:07.229127    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lndx7\" (UniqueName: \"kubernetes.io/projected/6d0a7973-cc86-4fda-9056-11b5923bd14c-kube-api-access-lndx7\") pod \"network-check-target-4cb4f\" (UID: \"6d0a7973-cc86-4fda-9056-11b5923bd14c\") " pod="openshift-network-diagnostics/network-check-target-4cb4f"
Sep 17 17:42:07.381192 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:07.381164    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 17:42:07.410682 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:07.410660    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-lspl9\""
Sep 17 17:42:07.414455 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:07.414427    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4cb4f"
Sep 17 17:42:07.421407 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:07.421389    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-vjstm"]
Sep 17 17:42:07.423622 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:42:07.423598    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb73a8fe_d34c_41c8_8ba4_8040b4bacf13.slice/crio-44e27ea90bc9f46f04059c14e4910d0a7cd281732207cd0da7571a2068b8b75b WatchSource:0}: Error finding container 44e27ea90bc9f46f04059c14e4910d0a7cd281732207cd0da7571a2068b8b75b: Status 404 returned error can't find the container with id 44e27ea90bc9f46f04059c14e4910d0a7cd281732207cd0da7571a2068b8b75b
Sep 17 17:42:07.442909 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:07.442886    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-564fd7cf7d-hwkkx"]
Sep 17 17:42:07.453982 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:42:07.453958    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ab7aae3_053b_4e36_9c4a_c523f0cfb615.slice/crio-4a1d99d291c518cc9c208ab1af1f45e20d5d29d3aee9267c432295868a8b1a76 WatchSource:0}: Error finding container 4a1d99d291c518cc9c208ab1af1f45e20d5d29d3aee9267c432295868a8b1a76: Status 404 returned error can't find the container with id 4a1d99d291c518cc9c208ab1af1f45e20d5d29d3aee9267c432295868a8b1a76
Sep 17 17:42:07.466767 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:07.466742    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-xg2wr"]
Sep 17 17:42:07.466920 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:07.466884    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-564fd7cf7d-hwkkx"
Sep 17 17:42:07.468247 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:07.468141    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-564fd7cf7d-hwkkx"]
Sep 17 17:42:07.474245 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:07.474138    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Sep 17 17:42:07.475042 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:07.474992    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Sep 17 17:42:07.476869 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:07.476254    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-68mmhgr33v1q2\""
Sep 17 17:42:07.476869 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:07.476475    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Sep 17 17:42:07.476869 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:07.476705    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Sep 17 17:42:07.477062 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:07.477035    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Sep 17 17:42:07.477559 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:07.477369    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-5clwz\""
Sep 17 17:42:07.539795 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:07.539767    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 17:42:07.602026 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:07.601994    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/38d1179d-e6c3-4949-bf01-b6ed37451005-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-564fd7cf7d-hwkkx\" (UID: \"38d1179d-e6c3-4949-bf01-b6ed37451005\") " pod="openshift-monitoring/thanos-querier-564fd7cf7d-hwkkx"
Sep 17 17:42:07.602026 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:07.602024    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/38d1179d-e6c3-4949-bf01-b6ed37451005-secret-grpc-tls\") pod \"thanos-querier-564fd7cf7d-hwkkx\" (UID: \"38d1179d-e6c3-4949-bf01-b6ed37451005\") " pod="openshift-monitoring/thanos-querier-564fd7cf7d-hwkkx"
Sep 17 17:42:07.602205 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:07.602043    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/38d1179d-e6c3-4949-bf01-b6ed37451005-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-564fd7cf7d-hwkkx\" (UID: \"38d1179d-e6c3-4949-bf01-b6ed37451005\") " pod="openshift-monitoring/thanos-querier-564fd7cf7d-hwkkx"
Sep 17 17:42:07.602205 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:07.602060    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/38d1179d-e6c3-4949-bf01-b6ed37451005-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-564fd7cf7d-hwkkx\" (UID: \"38d1179d-e6c3-4949-bf01-b6ed37451005\") " pod="openshift-monitoring/thanos-querier-564fd7cf7d-hwkkx"
Sep 17 17:42:07.602205 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:07.602108    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fs5cj\" (UniqueName: \"kubernetes.io/projected/38d1179d-e6c3-4949-bf01-b6ed37451005-kube-api-access-fs5cj\") pod \"thanos-querier-564fd7cf7d-hwkkx\" (UID: \"38d1179d-e6c3-4949-bf01-b6ed37451005\") " pod="openshift-monitoring/thanos-querier-564fd7cf7d-hwkkx"
Sep 17 17:42:07.602205 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:07.602129    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/38d1179d-e6c3-4949-bf01-b6ed37451005-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-564fd7cf7d-hwkkx\" (UID: \"38d1179d-e6c3-4949-bf01-b6ed37451005\") " pod="openshift-monitoring/thanos-querier-564fd7cf7d-hwkkx"
Sep 17 17:42:07.602205 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:07.602161    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/38d1179d-e6c3-4949-bf01-b6ed37451005-metrics-client-ca\") pod \"thanos-querier-564fd7cf7d-hwkkx\" (UID: \"38d1179d-e6c3-4949-bf01-b6ed37451005\") " pod="openshift-monitoring/thanos-querier-564fd7cf7d-hwkkx"
Sep 17 17:42:07.602205 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:07.602182    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/38d1179d-e6c3-4949-bf01-b6ed37451005-secret-thanos-querier-tls\") pod \"thanos-querier-564fd7cf7d-hwkkx\" (UID: \"38d1179d-e6c3-4949-bf01-b6ed37451005\") " pod="openshift-monitoring/thanos-querier-564fd7cf7d-hwkkx"
Sep 17 17:42:07.669507 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:07.669469    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-84c885658c-wg927" event={"ID":"db8f376a-5499-4162-b13f-f552de771aae","Type":"ContainerStarted","Data":"1c7b0fa36643225939c26911c3cf9d6c0bdc2d0ceb002699b6f10710227286c3"}
Sep 17 17:42:07.669699 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:07.669553    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-84c885658c-wg927" event={"ID":"db8f376a-5499-4162-b13f-f552de771aae","Type":"ContainerStarted","Data":"bdce435eac99e33fee21615e80c33ca4585bcaf6ad33cd5e028fda4116153370"}
Sep 17 17:42:07.669699 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:07.669588    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-84c885658c-wg927" event={"ID":"db8f376a-5499-4162-b13f-f552de771aae","Type":"ContainerStarted","Data":"d1237df25d8c19ca65c9a4ddfb86a9173fef79afedc11b20c325a1ae6c2a9505"}
Sep 17 17:42:07.670775 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:07.670750    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-xg2wr" event={"ID":"5ab7aae3-053b-4e36-9c4a-c523f0cfb615","Type":"ContainerStarted","Data":"4a1d99d291c518cc9c208ab1af1f45e20d5d29d3aee9267c432295868a8b1a76"}
Sep 17 17:42:07.671986 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:07.671958    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-vjstm" event={"ID":"db73a8fe-d34c-41c8-8ba4-8040b4bacf13","Type":"ContainerStarted","Data":"44e27ea90bc9f46f04059c14e4910d0a7cd281732207cd0da7571a2068b8b75b"}
Sep 17 17:42:07.703097 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:07.703068    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fs5cj\" (UniqueName: \"kubernetes.io/projected/38d1179d-e6c3-4949-bf01-b6ed37451005-kube-api-access-fs5cj\") pod \"thanos-querier-564fd7cf7d-hwkkx\" (UID: \"38d1179d-e6c3-4949-bf01-b6ed37451005\") " pod="openshift-monitoring/thanos-querier-564fd7cf7d-hwkkx"
Sep 17 17:42:07.703221 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:07.703108    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/38d1179d-e6c3-4949-bf01-b6ed37451005-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-564fd7cf7d-hwkkx\" (UID: \"38d1179d-e6c3-4949-bf01-b6ed37451005\") " pod="openshift-monitoring/thanos-querier-564fd7cf7d-hwkkx"
Sep 17 17:42:07.703221 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:07.703142    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/38d1179d-e6c3-4949-bf01-b6ed37451005-metrics-client-ca\") pod \"thanos-querier-564fd7cf7d-hwkkx\" (UID: \"38d1179d-e6c3-4949-bf01-b6ed37451005\") " pod="openshift-monitoring/thanos-querier-564fd7cf7d-hwkkx"
Sep 17 17:42:07.703221 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:07.703171    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/38d1179d-e6c3-4949-bf01-b6ed37451005-secret-thanos-querier-tls\") pod \"thanos-querier-564fd7cf7d-hwkkx\" (UID: \"38d1179d-e6c3-4949-bf01-b6ed37451005\") " pod="openshift-monitoring/thanos-querier-564fd7cf7d-hwkkx"
Sep 17 17:42:07.703382 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:07.703246    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/38d1179d-e6c3-4949-bf01-b6ed37451005-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-564fd7cf7d-hwkkx\" (UID: \"38d1179d-e6c3-4949-bf01-b6ed37451005\") " pod="openshift-monitoring/thanos-querier-564fd7cf7d-hwkkx"
Sep 17 17:42:07.703382 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:07.703272    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/38d1179d-e6c3-4949-bf01-b6ed37451005-secret-grpc-tls\") pod \"thanos-querier-564fd7cf7d-hwkkx\" (UID: \"38d1179d-e6c3-4949-bf01-b6ed37451005\") " pod="openshift-monitoring/thanos-querier-564fd7cf7d-hwkkx"
Sep 17 17:42:07.703382 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:07.703297    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/38d1179d-e6c3-4949-bf01-b6ed37451005-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-564fd7cf7d-hwkkx\" (UID: \"38d1179d-e6c3-4949-bf01-b6ed37451005\") " pod="openshift-monitoring/thanos-querier-564fd7cf7d-hwkkx"
Sep 17 17:42:07.703382 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:07.703327    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/38d1179d-e6c3-4949-bf01-b6ed37451005-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-564fd7cf7d-hwkkx\" (UID: \"38d1179d-e6c3-4949-bf01-b6ed37451005\") " pod="openshift-monitoring/thanos-querier-564fd7cf7d-hwkkx"
Sep 17 17:42:07.703991 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:07.703966    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/38d1179d-e6c3-4949-bf01-b6ed37451005-metrics-client-ca\") pod \"thanos-querier-564fd7cf7d-hwkkx\" (UID: \"38d1179d-e6c3-4949-bf01-b6ed37451005\") " pod="openshift-monitoring/thanos-querier-564fd7cf7d-hwkkx"
Sep 17 17:42:07.705788 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:07.705763    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/38d1179d-e6c3-4949-bf01-b6ed37451005-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-564fd7cf7d-hwkkx\" (UID: \"38d1179d-e6c3-4949-bf01-b6ed37451005\") " pod="openshift-monitoring/thanos-querier-564fd7cf7d-hwkkx"
Sep 17 17:42:07.705920 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:07.705899    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/38d1179d-e6c3-4949-bf01-b6ed37451005-secret-grpc-tls\") pod \"thanos-querier-564fd7cf7d-hwkkx\" (UID: \"38d1179d-e6c3-4949-bf01-b6ed37451005\") " pod="openshift-monitoring/thanos-querier-564fd7cf7d-hwkkx"
Sep 17 17:42:07.706078 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:07.706056    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/38d1179d-e6c3-4949-bf01-b6ed37451005-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-564fd7cf7d-hwkkx\" (UID: \"38d1179d-e6c3-4949-bf01-b6ed37451005\") " pod="openshift-monitoring/thanos-querier-564fd7cf7d-hwkkx"
Sep 17 17:42:07.706158 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:07.706142    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/38d1179d-e6c3-4949-bf01-b6ed37451005-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-564fd7cf7d-hwkkx\" (UID: \"38d1179d-e6c3-4949-bf01-b6ed37451005\") " pod="openshift-monitoring/thanos-querier-564fd7cf7d-hwkkx"
Sep 17 17:42:07.706215 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:07.706189    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/38d1179d-e6c3-4949-bf01-b6ed37451005-secret-thanos-querier-tls\") pod \"thanos-querier-564fd7cf7d-hwkkx\" (UID: \"38d1179d-e6c3-4949-bf01-b6ed37451005\") " pod="openshift-monitoring/thanos-querier-564fd7cf7d-hwkkx"
Sep 17 17:42:07.706275 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:07.706254    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/38d1179d-e6c3-4949-bf01-b6ed37451005-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-564fd7cf7d-hwkkx\" (UID: \"38d1179d-e6c3-4949-bf01-b6ed37451005\") " pod="openshift-monitoring/thanos-querier-564fd7cf7d-hwkkx"
Sep 17 17:42:07.720291 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:07.720243    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fs5cj\" (UniqueName: \"kubernetes.io/projected/38d1179d-e6c3-4949-bf01-b6ed37451005-kube-api-access-fs5cj\") pod \"thanos-querier-564fd7cf7d-hwkkx\" (UID: \"38d1179d-e6c3-4949-bf01-b6ed37451005\") " pod="openshift-monitoring/thanos-querier-564fd7cf7d-hwkkx"
Sep 17 17:42:07.766626 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:07.766594    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 17:42:07.778680 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:07.778651    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-564fd7cf7d-hwkkx"
Sep 17 17:42:07.873145 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:42:07.873110    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d0a7973_cc86_4fda_9056_11b5923bd14c.slice/crio-eeccedfe013b077a9b90e18d5952f7e40eed483e7e89d545fa17727c537a631d WatchSource:0}: Error finding container eeccedfe013b077a9b90e18d5952f7e40eed483e7e89d545fa17727c537a631d: Status 404 returned error can't find the container with id eeccedfe013b077a9b90e18d5952f7e40eed483e7e89d545fa17727c537a631d
Sep 17 17:42:07.877268 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:07.877228    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-84c885658c-wg927" podStartSLOduration=2.341962503 podStartE2EDuration="3.877211983s" totalImagesPullingTime="1.53524948s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:42:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:42:07.74325742 +0000 UTC m=+68.766862555" watchObservedRunningTime="2026-09-17 17:42:07.877211983 +0000 UTC m=+68.900817118"
Sep 17 17:42:07.877867 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:07.877850    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-4cb4f"]
Sep 17 17:42:08.001093 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:08.000992    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 17:42:08.097745 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:08.097373    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-564fd7cf7d-hwkkx"]
Sep 17 17:42:08.100483 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:42:08.100448    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38d1179d_e6c3_4949_bf01_b6ed37451005.slice/crio-d76ea7c3333c1e1a5c40d7c604693c809b5f7876feb252e7d0a667397782b96c WatchSource:0}: Error finding container d76ea7c3333c1e1a5c40d7c604693c809b5f7876feb252e7d0a667397782b96c: Status 404 returned error can't find the container with id d76ea7c3333c1e1a5c40d7c604693c809b5f7876feb252e7d0a667397782b96c
Sep 17 17:42:08.281980 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:08.281859    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 17:42:08.394636 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:08.394569    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 17:42:08.679187 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:08.679075    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-564fd7cf7d-hwkkx" event={"ID":"38d1179d-e6c3-4949-bf01-b6ed37451005","Type":"ContainerStarted","Data":"d76ea7c3333c1e1a5c40d7c604693c809b5f7876feb252e7d0a667397782b96c"}
Sep 17 17:42:08.680975 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:08.680918    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-4cb4f" event={"ID":"6d0a7973-cc86-4fda-9056-11b5923bd14c","Type":"ContainerStarted","Data":"eeccedfe013b077a9b90e18d5952f7e40eed483e7e89d545fa17727c537a631d"}
Sep 17 17:42:08.962609 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:08.962489    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 17:42:09.054740 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:09.054672    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 17:42:09.128993 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:09.128926    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 17:42:09.475585 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:09.474032    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-8584f8444-pjmjp"]
Sep 17 17:42:09.477957 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:09.476888    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-8584f8444-pjmjp"
Sep 17 17:42:09.483801 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:09.483771    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Sep 17 17:42:09.484215 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:09.483957    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Sep 17 17:42:09.484215 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:09.483863    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-krr4c\""
Sep 17 17:42:09.484215 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:09.483918    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Sep 17 17:42:09.484215 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:09.483958    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Sep 17 17:42:09.484476 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:09.483851    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Sep 17 17:42:09.492107 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:09.492089    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\""
Sep 17 17:42:09.498232 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:09.498213    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-8584f8444-pjmjp"]
Sep 17 17:42:09.519863 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:09.519733    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d9aa6259-82fb-4485-8fc8-4afdf2a74d29-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-8584f8444-pjmjp\" (UID: \"d9aa6259-82fb-4485-8fc8-4afdf2a74d29\") " pod="openshift-monitoring/telemeter-client-8584f8444-pjmjp"
Sep 17 17:42:09.519863 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:09.519778    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/d9aa6259-82fb-4485-8fc8-4afdf2a74d29-secret-telemeter-client\") pod \"telemeter-client-8584f8444-pjmjp\" (UID: \"d9aa6259-82fb-4485-8fc8-4afdf2a74d29\") " pod="openshift-monitoring/telemeter-client-8584f8444-pjmjp"
Sep 17 17:42:09.519863 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:09.519803    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d9aa6259-82fb-4485-8fc8-4afdf2a74d29-telemeter-trusted-ca-bundle\") pod \"telemeter-client-8584f8444-pjmjp\" (UID: \"d9aa6259-82fb-4485-8fc8-4afdf2a74d29\") " pod="openshift-monitoring/telemeter-client-8584f8444-pjmjp"
Sep 17 17:42:09.520071 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:09.519867    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jzmg2\" (UniqueName: \"kubernetes.io/projected/d9aa6259-82fb-4485-8fc8-4afdf2a74d29-kube-api-access-jzmg2\") pod \"telemeter-client-8584f8444-pjmjp\" (UID: \"d9aa6259-82fb-4485-8fc8-4afdf2a74d29\") " pod="openshift-monitoring/telemeter-client-8584f8444-pjmjp"
Sep 17 17:42:09.520071 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:09.519899    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d9aa6259-82fb-4485-8fc8-4afdf2a74d29-metrics-client-ca\") pod \"telemeter-client-8584f8444-pjmjp\" (UID: \"d9aa6259-82fb-4485-8fc8-4afdf2a74d29\") " pod="openshift-monitoring/telemeter-client-8584f8444-pjmjp"
Sep 17 17:42:09.520071 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:09.519920    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d9aa6259-82fb-4485-8fc8-4afdf2a74d29-serving-certs-ca-bundle\") pod \"telemeter-client-8584f8444-pjmjp\" (UID: \"d9aa6259-82fb-4485-8fc8-4afdf2a74d29\") " pod="openshift-monitoring/telemeter-client-8584f8444-pjmjp"
Sep 17 17:42:09.520071 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:09.519940    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/d9aa6259-82fb-4485-8fc8-4afdf2a74d29-federate-client-tls\") pod \"telemeter-client-8584f8444-pjmjp\" (UID: \"d9aa6259-82fb-4485-8fc8-4afdf2a74d29\") " pod="openshift-monitoring/telemeter-client-8584f8444-pjmjp"
Sep 17 17:42:09.520071 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:09.519963    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/d9aa6259-82fb-4485-8fc8-4afdf2a74d29-telemeter-client-tls\") pod \"telemeter-client-8584f8444-pjmjp\" (UID: \"d9aa6259-82fb-4485-8fc8-4afdf2a74d29\") " pod="openshift-monitoring/telemeter-client-8584f8444-pjmjp"
Sep 17 17:42:09.621192 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:09.621153    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/d9aa6259-82fb-4485-8fc8-4afdf2a74d29-secret-telemeter-client\") pod \"telemeter-client-8584f8444-pjmjp\" (UID: \"d9aa6259-82fb-4485-8fc8-4afdf2a74d29\") " pod="openshift-monitoring/telemeter-client-8584f8444-pjmjp"
Sep 17 17:42:09.621373 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:09.621201    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d9aa6259-82fb-4485-8fc8-4afdf2a74d29-telemeter-trusted-ca-bundle\") pod \"telemeter-client-8584f8444-pjmjp\" (UID: \"d9aa6259-82fb-4485-8fc8-4afdf2a74d29\") " pod="openshift-monitoring/telemeter-client-8584f8444-pjmjp"
Sep 17 17:42:09.621373 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:09.621252    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jzmg2\" (UniqueName: \"kubernetes.io/projected/d9aa6259-82fb-4485-8fc8-4afdf2a74d29-kube-api-access-jzmg2\") pod \"telemeter-client-8584f8444-pjmjp\" (UID: \"d9aa6259-82fb-4485-8fc8-4afdf2a74d29\") " pod="openshift-monitoring/telemeter-client-8584f8444-pjmjp"
Sep 17 17:42:09.621373 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:09.621284    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d9aa6259-82fb-4485-8fc8-4afdf2a74d29-metrics-client-ca\") pod \"telemeter-client-8584f8444-pjmjp\" (UID: \"d9aa6259-82fb-4485-8fc8-4afdf2a74d29\") " pod="openshift-monitoring/telemeter-client-8584f8444-pjmjp"
Sep 17 17:42:09.621373 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:09.621319    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d9aa6259-82fb-4485-8fc8-4afdf2a74d29-serving-certs-ca-bundle\") pod \"telemeter-client-8584f8444-pjmjp\" (UID: \"d9aa6259-82fb-4485-8fc8-4afdf2a74d29\") " pod="openshift-monitoring/telemeter-client-8584f8444-pjmjp"
Sep 17 17:42:09.621373 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:09.621340    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/d9aa6259-82fb-4485-8fc8-4afdf2a74d29-federate-client-tls\") pod \"telemeter-client-8584f8444-pjmjp\" (UID: \"d9aa6259-82fb-4485-8fc8-4afdf2a74d29\") " pod="openshift-monitoring/telemeter-client-8584f8444-pjmjp"
Sep 17 17:42:09.621658 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:09.621374    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/d9aa6259-82fb-4485-8fc8-4afdf2a74d29-telemeter-client-tls\") pod \"telemeter-client-8584f8444-pjmjp\" (UID: \"d9aa6259-82fb-4485-8fc8-4afdf2a74d29\") " pod="openshift-monitoring/telemeter-client-8584f8444-pjmjp"
Sep 17 17:42:09.621658 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:09.621425    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d9aa6259-82fb-4485-8fc8-4afdf2a74d29-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-8584f8444-pjmjp\" (UID: \"d9aa6259-82fb-4485-8fc8-4afdf2a74d29\") " pod="openshift-monitoring/telemeter-client-8584f8444-pjmjp"
Sep 17 17:42:09.622327 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:09.622296    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d9aa6259-82fb-4485-8fc8-4afdf2a74d29-metrics-client-ca\") pod \"telemeter-client-8584f8444-pjmjp\" (UID: \"d9aa6259-82fb-4485-8fc8-4afdf2a74d29\") " pod="openshift-monitoring/telemeter-client-8584f8444-pjmjp"
Sep 17 17:42:09.622709 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:09.622680    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d9aa6259-82fb-4485-8fc8-4afdf2a74d29-telemeter-trusted-ca-bundle\") pod \"telemeter-client-8584f8444-pjmjp\" (UID: \"d9aa6259-82fb-4485-8fc8-4afdf2a74d29\") " pod="openshift-monitoring/telemeter-client-8584f8444-pjmjp"
Sep 17 17:42:09.623267 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:09.623237    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d9aa6259-82fb-4485-8fc8-4afdf2a74d29-serving-certs-ca-bundle\") pod \"telemeter-client-8584f8444-pjmjp\" (UID: \"d9aa6259-82fb-4485-8fc8-4afdf2a74d29\") " pod="openshift-monitoring/telemeter-client-8584f8444-pjmjp"
Sep 17 17:42:09.624442 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:09.624393    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d9aa6259-82fb-4485-8fc8-4afdf2a74d29-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-8584f8444-pjmjp\" (UID: \"d9aa6259-82fb-4485-8fc8-4afdf2a74d29\") " pod="openshift-monitoring/telemeter-client-8584f8444-pjmjp"
Sep 17 17:42:09.624859 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:09.624796    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/d9aa6259-82fb-4485-8fc8-4afdf2a74d29-secret-telemeter-client\") pod \"telemeter-client-8584f8444-pjmjp\" (UID: \"d9aa6259-82fb-4485-8fc8-4afdf2a74d29\") " pod="openshift-monitoring/telemeter-client-8584f8444-pjmjp"
Sep 17 17:42:09.625327 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:09.625261    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/d9aa6259-82fb-4485-8fc8-4afdf2a74d29-federate-client-tls\") pod \"telemeter-client-8584f8444-pjmjp\" (UID: \"d9aa6259-82fb-4485-8fc8-4afdf2a74d29\") " pod="openshift-monitoring/telemeter-client-8584f8444-pjmjp"
Sep 17 17:42:09.625673 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:09.625649    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/d9aa6259-82fb-4485-8fc8-4afdf2a74d29-telemeter-client-tls\") pod \"telemeter-client-8584f8444-pjmjp\" (UID: \"d9aa6259-82fb-4485-8fc8-4afdf2a74d29\") " pod="openshift-monitoring/telemeter-client-8584f8444-pjmjp"
Sep 17 17:42:09.635376 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:09.635304    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jzmg2\" (UniqueName: \"kubernetes.io/projected/d9aa6259-82fb-4485-8fc8-4afdf2a74d29-kube-api-access-jzmg2\") pod \"telemeter-client-8584f8444-pjmjp\" (UID: \"d9aa6259-82fb-4485-8fc8-4afdf2a74d29\") " pod="openshift-monitoring/telemeter-client-8584f8444-pjmjp"
Sep 17 17:42:09.689048 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:09.688948    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-xg2wr" event={"ID":"5ab7aae3-053b-4e36-9c4a-c523f0cfb615","Type":"ContainerStarted","Data":"33b1d066c14666d8d82c009d6f369189e45f9eb827cfbe9a112b279cfa531854"}
Sep 17 17:42:09.689199 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:09.689064    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-xg2wr" event={"ID":"5ab7aae3-053b-4e36-9c4a-c523f0cfb615","Type":"ContainerStarted","Data":"62a5b66b24dd8ae61b76730ee0f31725d14036ba86ff13301bd2890c152fe6b6"}
Sep 17 17:42:09.790738 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:09.790638    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-8584f8444-pjmjp"
Sep 17 17:42:09.868057 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:09.867942    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 17:42:09.966015 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:09.965979    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 17:42:10.611122 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:10.611063    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-xg2wr" podStartSLOduration=68.42180397 podStartE2EDuration="1m9.611039985s" totalImagesPullingTime="1.189236015s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:41:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:42:09.715061644 +0000 UTC m=+70.738666805" watchObservedRunningTime="2026-09-17 17:42:10.611039985 +0000 UTC m=+71.634645123"
Sep 17 17:42:10.612161 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:10.612129    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 17:42:10.612593 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:10.612575    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 17 17:42:10.616478 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:10.615973    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:42:10.622283 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:10.621814    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 17 17:42:10.622283 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:10.622038    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 17 17:42:10.622761 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:10.622586    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Sep 17 17:42:10.623274 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:10.623122    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 17 17:42:10.623274 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:10.623155    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 17 17:42:10.623274 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:10.623164    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 17 17:42:10.623274 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:10.623190    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 17 17:42:10.623509 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:10.623316    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 17 17:42:10.623509 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:10.623328    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-dgttk\""
Sep 17 17:42:10.623509 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:10.623381    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-flrbqlhtmqr18\""
Sep 17 17:42:10.623509 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:10.623434    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 17 17:42:10.623727 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:10.623551    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 17 17:42:10.623727 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:10.623652    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 17 17:42:10.624794 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:10.624012    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 17 17:42:10.624794 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:10.624023    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 17 17:42:10.625631 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:10.625411    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 17 17:42:10.627196 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:10.626887    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 17 17:42:10.645882 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:10.645855    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 17 17:42:10.732487 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:10.732449    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/8dcb3c07-9ad2-4670-b801-0acce83c6faf-config\") pod \"prometheus-k8s-0\" (UID: \"8dcb3c07-9ad2-4670-b801-0acce83c6faf\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:42:10.732677 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:10.732510    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/8dcb3c07-9ad2-4670-b801-0acce83c6faf-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"8dcb3c07-9ad2-4670-b801-0acce83c6faf\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:42:10.732677 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:10.732586    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8dcb3c07-9ad2-4670-b801-0acce83c6faf-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"8dcb3c07-9ad2-4670-b801-0acce83c6faf\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:42:10.732677 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:10.732623    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8dcb3c07-9ad2-4670-b801-0acce83c6faf-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"8dcb3c07-9ad2-4670-b801-0acce83c6faf\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:42:10.732677 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:10.732670    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8dcb3c07-9ad2-4670-b801-0acce83c6faf-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"8dcb3c07-9ad2-4670-b801-0acce83c6faf\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:42:10.732887 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:10.732715    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8dcb3c07-9ad2-4670-b801-0acce83c6faf-config-out\") pod \"prometheus-k8s-0\" (UID: \"8dcb3c07-9ad2-4670-b801-0acce83c6faf\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:42:10.732887 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:10.732743    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/8dcb3c07-9ad2-4670-b801-0acce83c6faf-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"8dcb3c07-9ad2-4670-b801-0acce83c6faf\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:42:10.732887 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:10.732774    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8dcb3c07-9ad2-4670-b801-0acce83c6faf-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8dcb3c07-9ad2-4670-b801-0acce83c6faf\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:42:10.732887 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:10.732803    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/8dcb3c07-9ad2-4670-b801-0acce83c6faf-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"8dcb3c07-9ad2-4670-b801-0acce83c6faf\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:42:10.732887 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:10.732835    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8dcb3c07-9ad2-4670-b801-0acce83c6faf-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"8dcb3c07-9ad2-4670-b801-0acce83c6faf\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:42:10.733057 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:10.732919    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/8dcb3c07-9ad2-4670-b801-0acce83c6faf-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"8dcb3c07-9ad2-4670-b801-0acce83c6faf\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:42:10.733057 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:10.732959    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8dcb3c07-9ad2-4670-b801-0acce83c6faf-web-config\") pod \"prometheus-k8s-0\" (UID: \"8dcb3c07-9ad2-4670-b801-0acce83c6faf\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:42:10.733057 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:10.733014    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/8dcb3c07-9ad2-4670-b801-0acce83c6faf-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"8dcb3c07-9ad2-4670-b801-0acce83c6faf\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:42:10.733165 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:10.733095    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/8dcb3c07-9ad2-4670-b801-0acce83c6faf-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"8dcb3c07-9ad2-4670-b801-0acce83c6faf\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:42:10.733165 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:10.733129    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8dcb3c07-9ad2-4670-b801-0acce83c6faf-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8dcb3c07-9ad2-4670-b801-0acce83c6faf\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:42:10.733165 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:10.733158    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8dcb3c07-9ad2-4670-b801-0acce83c6faf-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"8dcb3c07-9ad2-4670-b801-0acce83c6faf\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:42:10.733279 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:10.733181    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8dcb3c07-9ad2-4670-b801-0acce83c6faf-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8dcb3c07-9ad2-4670-b801-0acce83c6faf\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:42:10.733279 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:10.733207    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/8dcb3c07-9ad2-4670-b801-0acce83c6faf-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"8dcb3c07-9ad2-4670-b801-0acce83c6faf\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:42:10.733279 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:10.733238    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/8dcb3c07-9ad2-4670-b801-0acce83c6faf-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"8dcb3c07-9ad2-4670-b801-0acce83c6faf\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:42:10.733279 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:10.733274    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6t448\" (UniqueName: \"kubernetes.io/projected/8dcb3c07-9ad2-4670-b801-0acce83c6faf-kube-api-access-6t448\") pod \"prometheus-k8s-0\" (UID: \"8dcb3c07-9ad2-4670-b801-0acce83c6faf\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:42:10.834413 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:10.834373    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/8dcb3c07-9ad2-4670-b801-0acce83c6faf-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"8dcb3c07-9ad2-4670-b801-0acce83c6faf\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:42:10.834603 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:10.834421    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8dcb3c07-9ad2-4670-b801-0acce83c6faf-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"8dcb3c07-9ad2-4670-b801-0acce83c6faf\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:42:10.834603 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:10.834448    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8dcb3c07-9ad2-4670-b801-0acce83c6faf-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"8dcb3c07-9ad2-4670-b801-0acce83c6faf\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:42:10.834603 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:10.834477    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8dcb3c07-9ad2-4670-b801-0acce83c6faf-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"8dcb3c07-9ad2-4670-b801-0acce83c6faf\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:42:10.834603 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:10.834507    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8dcb3c07-9ad2-4670-b801-0acce83c6faf-config-out\") pod \"prometheus-k8s-0\" (UID: \"8dcb3c07-9ad2-4670-b801-0acce83c6faf\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:42:10.834603 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:10.834523    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/8dcb3c07-9ad2-4670-b801-0acce83c6faf-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"8dcb3c07-9ad2-4670-b801-0acce83c6faf\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:42:10.834603 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:10.834563    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8dcb3c07-9ad2-4670-b801-0acce83c6faf-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8dcb3c07-9ad2-4670-b801-0acce83c6faf\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:42:10.834603 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:10.834593    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/8dcb3c07-9ad2-4670-b801-0acce83c6faf-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"8dcb3c07-9ad2-4670-b801-0acce83c6faf\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:42:10.834968 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:10.834621    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8dcb3c07-9ad2-4670-b801-0acce83c6faf-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"8dcb3c07-9ad2-4670-b801-0acce83c6faf\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:42:10.834968 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:10.834671    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/8dcb3c07-9ad2-4670-b801-0acce83c6faf-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"8dcb3c07-9ad2-4670-b801-0acce83c6faf\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:42:10.834968 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:10.834703    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8dcb3c07-9ad2-4670-b801-0acce83c6faf-web-config\") pod \"prometheus-k8s-0\" (UID: \"8dcb3c07-9ad2-4670-b801-0acce83c6faf\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:42:10.834968 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:10.834800    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/8dcb3c07-9ad2-4670-b801-0acce83c6faf-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"8dcb3c07-9ad2-4670-b801-0acce83c6faf\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:42:10.834968 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:10.834873    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/8dcb3c07-9ad2-4670-b801-0acce83c6faf-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"8dcb3c07-9ad2-4670-b801-0acce83c6faf\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:42:10.834968 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:10.834911    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8dcb3c07-9ad2-4670-b801-0acce83c6faf-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8dcb3c07-9ad2-4670-b801-0acce83c6faf\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:42:10.834968 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:10.834939    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8dcb3c07-9ad2-4670-b801-0acce83c6faf-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"8dcb3c07-9ad2-4670-b801-0acce83c6faf\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:42:10.834968 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:10.834969    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8dcb3c07-9ad2-4670-b801-0acce83c6faf-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8dcb3c07-9ad2-4670-b801-0acce83c6faf\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:42:10.835366 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:10.834998    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/8dcb3c07-9ad2-4670-b801-0acce83c6faf-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"8dcb3c07-9ad2-4670-b801-0acce83c6faf\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:42:10.835366 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:10.835023    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/8dcb3c07-9ad2-4670-b801-0acce83c6faf-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"8dcb3c07-9ad2-4670-b801-0acce83c6faf\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:42:10.835366 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:10.835047    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6t448\" (UniqueName: \"kubernetes.io/projected/8dcb3c07-9ad2-4670-b801-0acce83c6faf-kube-api-access-6t448\") pod \"prometheus-k8s-0\" (UID: \"8dcb3c07-9ad2-4670-b801-0acce83c6faf\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:42:10.835366 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:10.835078    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/8dcb3c07-9ad2-4670-b801-0acce83c6faf-config\") pod \"prometheus-k8s-0\" (UID: \"8dcb3c07-9ad2-4670-b801-0acce83c6faf\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:42:10.836245 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:10.835925    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8dcb3c07-9ad2-4670-b801-0acce83c6faf-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"8dcb3c07-9ad2-4670-b801-0acce83c6faf\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:42:10.836603 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:10.836579    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/8dcb3c07-9ad2-4670-b801-0acce83c6faf-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"8dcb3c07-9ad2-4670-b801-0acce83c6faf\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:42:10.838007 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:10.837981    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8dcb3c07-9ad2-4670-b801-0acce83c6faf-config-out\") pod \"prometheus-k8s-0\" (UID: \"8dcb3c07-9ad2-4670-b801-0acce83c6faf\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:42:10.838157 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:10.838128    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/8dcb3c07-9ad2-4670-b801-0acce83c6faf-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"8dcb3c07-9ad2-4670-b801-0acce83c6faf\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:42:10.839560 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:10.839222    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/8dcb3c07-9ad2-4670-b801-0acce83c6faf-config\") pod \"prometheus-k8s-0\" (UID: \"8dcb3c07-9ad2-4670-b801-0acce83c6faf\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:42:10.840590 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:10.839983    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8dcb3c07-9ad2-4670-b801-0acce83c6faf-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8dcb3c07-9ad2-4670-b801-0acce83c6faf\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:42:10.840590 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:10.840020    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8dcb3c07-9ad2-4670-b801-0acce83c6faf-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8dcb3c07-9ad2-4670-b801-0acce83c6faf\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:42:10.840590 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:10.840442    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8dcb3c07-9ad2-4670-b801-0acce83c6faf-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"8dcb3c07-9ad2-4670-b801-0acce83c6faf\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:42:10.840590 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:10.840478    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/8dcb3c07-9ad2-4670-b801-0acce83c6faf-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"8dcb3c07-9ad2-4670-b801-0acce83c6faf\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:42:10.840590 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:10.840502    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8dcb3c07-9ad2-4670-b801-0acce83c6faf-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"8dcb3c07-9ad2-4670-b801-0acce83c6faf\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:42:10.841070 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:10.841039    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/8dcb3c07-9ad2-4670-b801-0acce83c6faf-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"8dcb3c07-9ad2-4670-b801-0acce83c6faf\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:42:10.842905 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:10.842882    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8dcb3c07-9ad2-4670-b801-0acce83c6faf-web-config\") pod \"prometheus-k8s-0\" (UID: \"8dcb3c07-9ad2-4670-b801-0acce83c6faf\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:42:10.842905 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:10.842891    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/8dcb3c07-9ad2-4670-b801-0acce83c6faf-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"8dcb3c07-9ad2-4670-b801-0acce83c6faf\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:42:10.845034 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:10.843098    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8dcb3c07-9ad2-4670-b801-0acce83c6faf-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8dcb3c07-9ad2-4670-b801-0acce83c6faf\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:42:10.845034 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:10.843805    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/8dcb3c07-9ad2-4670-b801-0acce83c6faf-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"8dcb3c07-9ad2-4670-b801-0acce83c6faf\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:42:10.845034 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:10.844432    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8dcb3c07-9ad2-4670-b801-0acce83c6faf-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"8dcb3c07-9ad2-4670-b801-0acce83c6faf\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:42:10.845034 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:10.844891    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8dcb3c07-9ad2-4670-b801-0acce83c6faf-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"8dcb3c07-9ad2-4670-b801-0acce83c6faf\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:42:10.845243 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:10.845082    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/8dcb3c07-9ad2-4670-b801-0acce83c6faf-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"8dcb3c07-9ad2-4670-b801-0acce83c6faf\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:42:10.845767 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:10.845748    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/8dcb3c07-9ad2-4670-b801-0acce83c6faf-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"8dcb3c07-9ad2-4670-b801-0acce83c6faf\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:42:10.849190 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:10.849170    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6t448\" (UniqueName: \"kubernetes.io/projected/8dcb3c07-9ad2-4670-b801-0acce83c6faf-kube-api-access-6t448\") pod \"prometheus-k8s-0\" (UID: \"8dcb3c07-9ad2-4670-b801-0acce83c6faf\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:42:10.930105 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:10.930021    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:42:11.016373 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:11.016337    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-74b7bc945c-5f8mh"]
Sep 17 17:42:11.046315 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:11.046279    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-74f79d49bf-fjjc5"]
Sep 17 17:42:11.049060 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:11.048733    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-74f79d49bf-fjjc5"
Sep 17 17:42:11.065345 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:11.065315    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-74f79d49bf-fjjc5"]
Sep 17 17:42:11.138508 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:11.138465    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1c94b604-5a62-4fff-a6d4-a21c31c62da0-console-serving-cert\") pod \"console-74f79d49bf-fjjc5\" (UID: \"1c94b604-5a62-4fff-a6d4-a21c31c62da0\") " pod="openshift-console/console-74f79d49bf-fjjc5"
Sep 17 17:42:11.138685 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:11.138543    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1c94b604-5a62-4fff-a6d4-a21c31c62da0-console-oauth-config\") pod \"console-74f79d49bf-fjjc5\" (UID: \"1c94b604-5a62-4fff-a6d4-a21c31c62da0\") " pod="openshift-console/console-74f79d49bf-fjjc5"
Sep 17 17:42:11.138685 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:11.138616    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1c94b604-5a62-4fff-a6d4-a21c31c62da0-trusted-ca-bundle\") pod \"console-74f79d49bf-fjjc5\" (UID: \"1c94b604-5a62-4fff-a6d4-a21c31c62da0\") " pod="openshift-console/console-74f79d49bf-fjjc5"
Sep 17 17:42:11.138685 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:11.138649    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1c94b604-5a62-4fff-a6d4-a21c31c62da0-service-ca\") pod \"console-74f79d49bf-fjjc5\" (UID: \"1c94b604-5a62-4fff-a6d4-a21c31c62da0\") " pod="openshift-console/console-74f79d49bf-fjjc5"
Sep 17 17:42:11.138685 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:11.138680    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1c94b604-5a62-4fff-a6d4-a21c31c62da0-console-config\") pod \"console-74f79d49bf-fjjc5\" (UID: \"1c94b604-5a62-4fff-a6d4-a21c31c62da0\") " pod="openshift-console/console-74f79d49bf-fjjc5"
Sep 17 17:42:11.138907 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:11.138708    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bt4k2\" (UniqueName: \"kubernetes.io/projected/1c94b604-5a62-4fff-a6d4-a21c31c62da0-kube-api-access-bt4k2\") pod \"console-74f79d49bf-fjjc5\" (UID: \"1c94b604-5a62-4fff-a6d4-a21c31c62da0\") " pod="openshift-console/console-74f79d49bf-fjjc5"
Sep 17 17:42:11.138907 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:11.138766    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1c94b604-5a62-4fff-a6d4-a21c31c62da0-oauth-serving-cert\") pod \"console-74f79d49bf-fjjc5\" (UID: \"1c94b604-5a62-4fff-a6d4-a21c31c62da0\") " pod="openshift-console/console-74f79d49bf-fjjc5"
Sep 17 17:42:11.240276 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:11.240196    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1c94b604-5a62-4fff-a6d4-a21c31c62da0-console-serving-cert\") pod \"console-74f79d49bf-fjjc5\" (UID: \"1c94b604-5a62-4fff-a6d4-a21c31c62da0\") " pod="openshift-console/console-74f79d49bf-fjjc5"
Sep 17 17:42:11.240276 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:11.240244    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1c94b604-5a62-4fff-a6d4-a21c31c62da0-console-oauth-config\") pod \"console-74f79d49bf-fjjc5\" (UID: \"1c94b604-5a62-4fff-a6d4-a21c31c62da0\") " pod="openshift-console/console-74f79d49bf-fjjc5"
Sep 17 17:42:11.240492 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:11.240295    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1c94b604-5a62-4fff-a6d4-a21c31c62da0-trusted-ca-bundle\") pod \"console-74f79d49bf-fjjc5\" (UID: \"1c94b604-5a62-4fff-a6d4-a21c31c62da0\") " pod="openshift-console/console-74f79d49bf-fjjc5"
Sep 17 17:42:11.240492 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:11.240318    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1c94b604-5a62-4fff-a6d4-a21c31c62da0-service-ca\") pod \"console-74f79d49bf-fjjc5\" (UID: \"1c94b604-5a62-4fff-a6d4-a21c31c62da0\") " pod="openshift-console/console-74f79d49bf-fjjc5"
Sep 17 17:42:11.240492 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:11.240347    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1c94b604-5a62-4fff-a6d4-a21c31c62da0-console-config\") pod \"console-74f79d49bf-fjjc5\" (UID: \"1c94b604-5a62-4fff-a6d4-a21c31c62da0\") " pod="openshift-console/console-74f79d49bf-fjjc5"
Sep 17 17:42:11.240492 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:11.240369    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bt4k2\" (UniqueName: \"kubernetes.io/projected/1c94b604-5a62-4fff-a6d4-a21c31c62da0-kube-api-access-bt4k2\") pod \"console-74f79d49bf-fjjc5\" (UID: \"1c94b604-5a62-4fff-a6d4-a21c31c62da0\") " pod="openshift-console/console-74f79d49bf-fjjc5"
Sep 17 17:42:11.240492 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:11.240399    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1c94b604-5a62-4fff-a6d4-a21c31c62da0-oauth-serving-cert\") pod \"console-74f79d49bf-fjjc5\" (UID: \"1c94b604-5a62-4fff-a6d4-a21c31c62da0\") " pod="openshift-console/console-74f79d49bf-fjjc5"
Sep 17 17:42:11.241273 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:11.241234    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1c94b604-5a62-4fff-a6d4-a21c31c62da0-oauth-serving-cert\") pod \"console-74f79d49bf-fjjc5\" (UID: \"1c94b604-5a62-4fff-a6d4-a21c31c62da0\") " pod="openshift-console/console-74f79d49bf-fjjc5"
Sep 17 17:42:11.241389 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:11.241277    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1c94b604-5a62-4fff-a6d4-a21c31c62da0-console-config\") pod \"console-74f79d49bf-fjjc5\" (UID: \"1c94b604-5a62-4fff-a6d4-a21c31c62da0\") " pod="openshift-console/console-74f79d49bf-fjjc5"
Sep 17 17:42:11.241389 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:11.241309    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1c94b604-5a62-4fff-a6d4-a21c31c62da0-service-ca\") pod \"console-74f79d49bf-fjjc5\" (UID: \"1c94b604-5a62-4fff-a6d4-a21c31c62da0\") " pod="openshift-console/console-74f79d49bf-fjjc5"
Sep 17 17:42:11.241389 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:11.241280    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1c94b604-5a62-4fff-a6d4-a21c31c62da0-trusted-ca-bundle\") pod \"console-74f79d49bf-fjjc5\" (UID: \"1c94b604-5a62-4fff-a6d4-a21c31c62da0\") " pod="openshift-console/console-74f79d49bf-fjjc5"
Sep 17 17:42:11.243242 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:11.243218    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1c94b604-5a62-4fff-a6d4-a21c31c62da0-console-serving-cert\") pod \"console-74f79d49bf-fjjc5\" (UID: \"1c94b604-5a62-4fff-a6d4-a21c31c62da0\") " pod="openshift-console/console-74f79d49bf-fjjc5"
Sep 17 17:42:11.243360 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:11.243271    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1c94b604-5a62-4fff-a6d4-a21c31c62da0-console-oauth-config\") pod \"console-74f79d49bf-fjjc5\" (UID: \"1c94b604-5a62-4fff-a6d4-a21c31c62da0\") " pod="openshift-console/console-74f79d49bf-fjjc5"
Sep 17 17:42:11.251294 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:11.251266    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bt4k2\" (UniqueName: \"kubernetes.io/projected/1c94b604-5a62-4fff-a6d4-a21c31c62da0-kube-api-access-bt4k2\") pod \"console-74f79d49bf-fjjc5\" (UID: \"1c94b604-5a62-4fff-a6d4-a21c31c62da0\") " pod="openshift-console/console-74f79d49bf-fjjc5"
Sep 17 17:42:11.361372 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:11.361340    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-74f79d49bf-fjjc5"
Sep 17 17:42:13.229854 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:13.229714    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-74f79d49bf-fjjc5"]
Sep 17 17:42:13.232653 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:42:13.232627    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c94b604_5a62_4fff_a6d4_a21c31c62da0.slice/crio-b498c6797110612bfccd4c8c90e106e47708d516879fa0181b96b211525f3199 WatchSource:0}: Error finding container b498c6797110612bfccd4c8c90e106e47708d516879fa0181b96b211525f3199: Status 404 returned error can't find the container with id b498c6797110612bfccd4c8c90e106e47708d516879fa0181b96b211525f3199
Sep 17 17:42:13.258340 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:13.258316    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-8584f8444-pjmjp"]
Sep 17 17:42:13.268606 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:42:13.268584    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9aa6259_82fb_4485_8fc8_4afdf2a74d29.slice/crio-660d31de547a7bade0dc6aad7a9763cc465c01b41694a5de66029394b9bf07e9 WatchSource:0}: Error finding container 660d31de547a7bade0dc6aad7a9763cc465c01b41694a5de66029394b9bf07e9: Status 404 returned error can't find the container with id 660d31de547a7bade0dc6aad7a9763cc465c01b41694a5de66029394b9bf07e9
Sep 17 17:42:13.324837 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:13.324810    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 17 17:42:13.327811 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:42:13.327787    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8dcb3c07_9ad2_4670_b801_0acce83c6faf.slice/crio-ad2b95b28324b1cb5ff2941e177dc3e52514f505c2801fd30c1df2bea3e2b603 WatchSource:0}: Error finding container ad2b95b28324b1cb5ff2941e177dc3e52514f505c2801fd30c1df2bea3e2b603: Status 404 returned error can't find the container with id ad2b95b28324b1cb5ff2941e177dc3e52514f505c2801fd30c1df2bea3e2b603
Sep 17 17:42:13.713595 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:13.713556    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-vjstm" event={"ID":"db73a8fe-d34c-41c8-8ba4-8040b4bacf13","Type":"ContainerStarted","Data":"f9e4d90a3d739392ab27d445e68d253e04a5361d4e03702bb4e64b5c28cac162"}
Sep 17 17:42:13.715941 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:13.715860    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-74f79d49bf-fjjc5" event={"ID":"1c94b604-5a62-4fff-a6d4-a21c31c62da0","Type":"ContainerStarted","Data":"3057c7ce83e1789950d56af28b4c261f7a844b21457a9b0cf453e8b86136d040"}
Sep 17 17:42:13.715941 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:13.715910    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-74f79d49bf-fjjc5" event={"ID":"1c94b604-5a62-4fff-a6d4-a21c31c62da0","Type":"ContainerStarted","Data":"b498c6797110612bfccd4c8c90e106e47708d516879fa0181b96b211525f3199"}
Sep 17 17:42:13.719696 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:13.719509    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-564fd7cf7d-hwkkx" event={"ID":"38d1179d-e6c3-4949-bf01-b6ed37451005","Type":"ContainerStarted","Data":"1da45cb3b9fa9fdb33f56eab633f3c748de43af5ca5fd122ea8b706c1572448d"}
Sep 17 17:42:13.719696 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:13.719603    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-564fd7cf7d-hwkkx" event={"ID":"38d1179d-e6c3-4949-bf01-b6ed37451005","Type":"ContainerStarted","Data":"ed227d0e375c63612f2dd20030a4d02ed5534f44e546841edbd755fd5c34a365"}
Sep 17 17:42:13.719696 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:13.719649    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-564fd7cf7d-hwkkx" event={"ID":"38d1179d-e6c3-4949-bf01-b6ed37451005","Type":"ContainerStarted","Data":"aaa941c0f8dcfa57a8c230db8829f59eadbdd8c70a01e5c43b63279bcdd39648"}
Sep 17 17:42:13.721130 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:13.721102    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8dcb3c07-9ad2-4670-b801-0acce83c6faf","Type":"ContainerStarted","Data":"ad2b95b28324b1cb5ff2941e177dc3e52514f505c2801fd30c1df2bea3e2b603"}
Sep 17 17:42:13.722603 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:13.722574    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-8584f8444-pjmjp" event={"ID":"d9aa6259-82fb-4485-8fc8-4afdf2a74d29","Type":"ContainerStarted","Data":"660d31de547a7bade0dc6aad7a9763cc465c01b41694a5de66029394b9bf07e9"}
Sep 17 17:42:13.724393 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:13.724365    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-4cb4f" event={"ID":"6d0a7973-cc86-4fda-9056-11b5923bd14c","Type":"ContainerStarted","Data":"af2d72b356a06825976ed6d790c9cb2c868d1553ceb412111bc80fd7c981fbb7"}
Sep 17 17:42:13.724592 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:13.724526    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-4cb4f"
Sep 17 17:42:13.759194 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:13.759089    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-vjstm" podStartSLOduration=67.252009123 podStartE2EDuration="1m12.759069884s" totalImagesPullingTime="5.507060761s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:41:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:42:13.740616585 +0000 UTC m=+74.764221722" watchObservedRunningTime="2026-09-17 17:42:13.759069884 +0000 UTC m=+74.782675021"
Sep 17 17:42:13.759393 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:13.759365    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-74f79d49bf-fjjc5" podStartSLOduration=2.75935575 podStartE2EDuration="2.75935575s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:42:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:42:13.758871365 +0000 UTC m=+74.782476522" watchObservedRunningTime="2026-09-17 17:42:13.75935575 +0000 UTC m=+74.782960880"
Sep 17 17:42:13.778898 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:13.778845    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-4cb4f" podStartSLOduration=67.723762268 podStartE2EDuration="1m12.778827656s" totalImagesPullingTime="5.055065388s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:41:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:42:13.777640726 +0000 UTC m=+74.801245878" watchObservedRunningTime="2026-09-17 17:42:13.778827656 +0000 UTC m=+74.802432792"
Sep 17 17:42:14.306035 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:14.306002    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-74b7bc945c-5f8mh"
Sep 17 17:42:14.462546 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:14.462493    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 17:42:14.566743 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:14.566668    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 17:42:14.674205 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:14.674169    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 17:42:14.735132 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:14.735095    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-564fd7cf7d-hwkkx" event={"ID":"38d1179d-e6c3-4949-bf01-b6ed37451005","Type":"ContainerStarted","Data":"edd6cfc1836c557fd147b832b43c1dea7b565a68b3a1367a0c0360306095c90f"}
Sep 17 17:42:14.735299 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:14.735189    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-564fd7cf7d-hwkkx" event={"ID":"38d1179d-e6c3-4949-bf01-b6ed37451005","Type":"ContainerStarted","Data":"376b7b20f80e5bdcc58a11d8d945ba75dd2068399b46daca47ed0ac0047308a5"}
Sep 17 17:42:14.735299 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:14.735229    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-564fd7cf7d-hwkkx" event={"ID":"38d1179d-e6c3-4949-bf01-b6ed37451005","Type":"ContainerStarted","Data":"edff64531800ab297adfa76e6f6d886b7b3ec4f8116777cc697061635302b868"}
Sep 17 17:42:14.735432 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:14.735369    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-564fd7cf7d-hwkkx"
Sep 17 17:42:14.736648 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:14.736625    2567 generic.go:356] "Generic (PLEG): container finished" podID="8dcb3c07-9ad2-4670-b801-0acce83c6faf" containerID="3d98a78d4042469903bb7f4870086e12672504526d45866281542e69841caf04" exitCode=0
Sep 17 17:42:14.736745 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:14.736718    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8dcb3c07-9ad2-4670-b801-0acce83c6faf","Type":"ContainerDied","Data":"3d98a78d4042469903bb7f4870086e12672504526d45866281542e69841caf04"}
Sep 17 17:42:14.778491 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:14.778466    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 17:42:14.811747 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:14.811701    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-564fd7cf7d-hwkkx" podStartSLOduration=2.379547339 podStartE2EDuration="7.811679133s" totalImagesPullingTime="5.432131794s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:42:07 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:42:14.769600424 +0000 UTC m=+75.793205582" watchObservedRunningTime="2026-09-17 17:42:14.811679133 +0000 UTC m=+75.835284271"
Sep 17 17:42:15.744408 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:15.744323    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-8584f8444-pjmjp" event={"ID":"d9aa6259-82fb-4485-8fc8-4afdf2a74d29","Type":"ContainerStarted","Data":"04b37efb21ed00f47e33548bd707cf7bfff9d7e3556b722e054c44e65c4554d8"}
Sep 17 17:42:15.744408 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:15.744394    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-8584f8444-pjmjp" event={"ID":"d9aa6259-82fb-4485-8fc8-4afdf2a74d29","Type":"ContainerStarted","Data":"6378260a17a8f63bd6e98bad604e0b6b8b259503ffeb82f14f8691803e3cb6fc"}
Sep 17 17:42:15.744924 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:15.744427    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-8584f8444-pjmjp" event={"ID":"d9aa6259-82fb-4485-8fc8-4afdf2a74d29","Type":"ContainerStarted","Data":"d911a7d80001958be3ab1c6d6db553ee2e74608a776cf8224fec23af64326f2a"}
Sep 17 17:42:15.790920 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:15.790865    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-8584f8444-pjmjp" podStartSLOduration=4.708040849 podStartE2EDuration="6.790840436s" totalImagesPullingTime="2.082799587s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:42:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:42:15.782296978 +0000 UTC m=+76.805902114" watchObservedRunningTime="2026-09-17 17:42:15.790840436 +0000 UTC m=+76.814445578"
Sep 17 17:42:15.936389 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:15.936354    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-74f79d49bf-fjjc5"]
Sep 17 17:42:15.981943 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:15.981908    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-c8d8d9b9c-h5mfl"]
Sep 17 17:42:15.984643 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:15.984609    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-c8d8d9b9c-h5mfl"
Sep 17 17:42:15.999661 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:15.999611    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-c8d8d9b9c-h5mfl"]
Sep 17 17:42:16.087071 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:16.087038    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3334a6ee-8f98-4a98-9351-1549e87d813d-service-ca\") pod \"console-c8d8d9b9c-h5mfl\" (UID: \"3334a6ee-8f98-4a98-9351-1549e87d813d\") " pod="openshift-console/console-c8d8d9b9c-h5mfl"
Sep 17 17:42:16.087249 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:16.087079    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3334a6ee-8f98-4a98-9351-1549e87d813d-trusted-ca-bundle\") pod \"console-c8d8d9b9c-h5mfl\" (UID: \"3334a6ee-8f98-4a98-9351-1549e87d813d\") " pod="openshift-console/console-c8d8d9b9c-h5mfl"
Sep 17 17:42:16.087249 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:16.087152    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3334a6ee-8f98-4a98-9351-1549e87d813d-console-serving-cert\") pod \"console-c8d8d9b9c-h5mfl\" (UID: \"3334a6ee-8f98-4a98-9351-1549e87d813d\") " pod="openshift-console/console-c8d8d9b9c-h5mfl"
Sep 17 17:42:16.087249 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:16.087231    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3334a6ee-8f98-4a98-9351-1549e87d813d-oauth-serving-cert\") pod \"console-c8d8d9b9c-h5mfl\" (UID: \"3334a6ee-8f98-4a98-9351-1549e87d813d\") " pod="openshift-console/console-c8d8d9b9c-h5mfl"
Sep 17 17:42:16.087406 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:16.087271    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3334a6ee-8f98-4a98-9351-1549e87d813d-console-oauth-config\") pod \"console-c8d8d9b9c-h5mfl\" (UID: \"3334a6ee-8f98-4a98-9351-1549e87d813d\") " pod="openshift-console/console-c8d8d9b9c-h5mfl"
Sep 17 17:42:16.087406 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:16.087305    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-729d4\" (UniqueName: \"kubernetes.io/projected/3334a6ee-8f98-4a98-9351-1549e87d813d-kube-api-access-729d4\") pod \"console-c8d8d9b9c-h5mfl\" (UID: \"3334a6ee-8f98-4a98-9351-1549e87d813d\") " pod="openshift-console/console-c8d8d9b9c-h5mfl"
Sep 17 17:42:16.087406 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:16.087360    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3334a6ee-8f98-4a98-9351-1549e87d813d-console-config\") pod \"console-c8d8d9b9c-h5mfl\" (UID: \"3334a6ee-8f98-4a98-9351-1549e87d813d\") " pod="openshift-console/console-c8d8d9b9c-h5mfl"
Sep 17 17:42:16.187898 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:16.187860    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3334a6ee-8f98-4a98-9351-1549e87d813d-service-ca\") pod \"console-c8d8d9b9c-h5mfl\" (UID: \"3334a6ee-8f98-4a98-9351-1549e87d813d\") " pod="openshift-console/console-c8d8d9b9c-h5mfl"
Sep 17 17:42:16.188059 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:16.187907    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3334a6ee-8f98-4a98-9351-1549e87d813d-trusted-ca-bundle\") pod \"console-c8d8d9b9c-h5mfl\" (UID: \"3334a6ee-8f98-4a98-9351-1549e87d813d\") " pod="openshift-console/console-c8d8d9b9c-h5mfl"
Sep 17 17:42:16.188059 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:16.187960    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3334a6ee-8f98-4a98-9351-1549e87d813d-console-serving-cert\") pod \"console-c8d8d9b9c-h5mfl\" (UID: \"3334a6ee-8f98-4a98-9351-1549e87d813d\") " pod="openshift-console/console-c8d8d9b9c-h5mfl"
Sep 17 17:42:16.188059 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:16.188001    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3334a6ee-8f98-4a98-9351-1549e87d813d-oauth-serving-cert\") pod \"console-c8d8d9b9c-h5mfl\" (UID: \"3334a6ee-8f98-4a98-9351-1549e87d813d\") " pod="openshift-console/console-c8d8d9b9c-h5mfl"
Sep 17 17:42:16.188059 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:16.188027    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3334a6ee-8f98-4a98-9351-1549e87d813d-console-oauth-config\") pod \"console-c8d8d9b9c-h5mfl\" (UID: \"3334a6ee-8f98-4a98-9351-1549e87d813d\") " pod="openshift-console/console-c8d8d9b9c-h5mfl"
Sep 17 17:42:16.188059 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:16.188057    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-729d4\" (UniqueName: \"kubernetes.io/projected/3334a6ee-8f98-4a98-9351-1549e87d813d-kube-api-access-729d4\") pod \"console-c8d8d9b9c-h5mfl\" (UID: \"3334a6ee-8f98-4a98-9351-1549e87d813d\") " pod="openshift-console/console-c8d8d9b9c-h5mfl"
Sep 17 17:42:16.188243 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:16.188087    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3334a6ee-8f98-4a98-9351-1549e87d813d-console-config\") pod \"console-c8d8d9b9c-h5mfl\" (UID: \"3334a6ee-8f98-4a98-9351-1549e87d813d\") " pod="openshift-console/console-c8d8d9b9c-h5mfl"
Sep 17 17:42:16.188716 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:16.188687    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3334a6ee-8f98-4a98-9351-1549e87d813d-service-ca\") pod \"console-c8d8d9b9c-h5mfl\" (UID: \"3334a6ee-8f98-4a98-9351-1549e87d813d\") " pod="openshift-console/console-c8d8d9b9c-h5mfl"
Sep 17 17:42:16.188928 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:16.188877    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3334a6ee-8f98-4a98-9351-1549e87d813d-console-config\") pod \"console-c8d8d9b9c-h5mfl\" (UID: \"3334a6ee-8f98-4a98-9351-1549e87d813d\") " pod="openshift-console/console-c8d8d9b9c-h5mfl"
Sep 17 17:42:16.189386 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:16.189331    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3334a6ee-8f98-4a98-9351-1549e87d813d-oauth-serving-cert\") pod \"console-c8d8d9b9c-h5mfl\" (UID: \"3334a6ee-8f98-4a98-9351-1549e87d813d\") " pod="openshift-console/console-c8d8d9b9c-h5mfl"
Sep 17 17:42:16.189752 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:16.189722    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3334a6ee-8f98-4a98-9351-1549e87d813d-trusted-ca-bundle\") pod \"console-c8d8d9b9c-h5mfl\" (UID: \"3334a6ee-8f98-4a98-9351-1549e87d813d\") " pod="openshift-console/console-c8d8d9b9c-h5mfl"
Sep 17 17:42:16.191338 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:16.191311    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3334a6ee-8f98-4a98-9351-1549e87d813d-console-oauth-config\") pod \"console-c8d8d9b9c-h5mfl\" (UID: \"3334a6ee-8f98-4a98-9351-1549e87d813d\") " pod="openshift-console/console-c8d8d9b9c-h5mfl"
Sep 17 17:42:16.191438 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:16.191415    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3334a6ee-8f98-4a98-9351-1549e87d813d-console-serving-cert\") pod \"console-c8d8d9b9c-h5mfl\" (UID: \"3334a6ee-8f98-4a98-9351-1549e87d813d\") " pod="openshift-console/console-c8d8d9b9c-h5mfl"
Sep 17 17:42:16.200129 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:16.200108    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-729d4\" (UniqueName: \"kubernetes.io/projected/3334a6ee-8f98-4a98-9351-1549e87d813d-kube-api-access-729d4\") pod \"console-c8d8d9b9c-h5mfl\" (UID: \"3334a6ee-8f98-4a98-9351-1549e87d813d\") " pod="openshift-console/console-c8d8d9b9c-h5mfl"
Sep 17 17:42:16.297251 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:16.297218    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-c8d8d9b9c-h5mfl"
Sep 17 17:42:16.585922 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:16.585897    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-c8d8d9b9c-h5mfl"]
Sep 17 17:42:16.590153 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:42:16.590105    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3334a6ee_8f98_4a98_9351_1549e87d813d.slice/crio-2cbff18c0663ff35243ea09b744aabd02ec2ef1fc571ba0b5fd0bec571a0c82c WatchSource:0}: Error finding container 2cbff18c0663ff35243ea09b744aabd02ec2ef1fc571ba0b5fd0bec571a0c82c: Status 404 returned error can't find the container with id 2cbff18c0663ff35243ea09b744aabd02ec2ef1fc571ba0b5fd0bec571a0c82c
Sep 17 17:42:16.755449 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:16.754982    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-c8d8d9b9c-h5mfl" event={"ID":"3334a6ee-8f98-4a98-9351-1549e87d813d","Type":"ContainerStarted","Data":"4137e52d1e08213c0ab6bde8aa32aeb03b3aeef1a349dc71557cd3b9f644b539"}
Sep 17 17:42:16.755449 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:16.755043    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-c8d8d9b9c-h5mfl" event={"ID":"3334a6ee-8f98-4a98-9351-1549e87d813d","Type":"ContainerStarted","Data":"2cbff18c0663ff35243ea09b744aabd02ec2ef1fc571ba0b5fd0bec571a0c82c"}
Sep 17 17:42:16.783380 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:16.783316    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-c8d8d9b9c-h5mfl" podStartSLOduration=1.783292827 podStartE2EDuration="1.783292827s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:42:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:42:16.780887809 +0000 UTC m=+77.804492970" watchObservedRunningTime="2026-09-17 17:42:16.783292827 +0000 UTC m=+77.806897964"
Sep 17 17:42:18.291725 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:18.291700    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 17:42:18.416955 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:18.416679    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 17:42:18.767735 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:18.767662    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8dcb3c07-9ad2-4670-b801-0acce83c6faf","Type":"ContainerStarted","Data":"d9543b6bf84dd3ea06ca00cc9480684d94b52827b927f3e39c5279872c7fbae1"}
Sep 17 17:42:18.767863 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:18.767752    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8dcb3c07-9ad2-4670-b801-0acce83c6faf","Type":"ContainerStarted","Data":"8c82f26264ba90782e9fb15f6fbd78bdb798f51c5a2391efa2e98a5f4fe00e43"}
Sep 17 17:42:18.767863 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:18.767795    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8dcb3c07-9ad2-4670-b801-0acce83c6faf","Type":"ContainerStarted","Data":"20126bf6b5656fe616b70ec294e7d7a76c3f6012919f3b40668a23c78ea35d40"}
Sep 17 17:42:18.767863 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:18.767825    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8dcb3c07-9ad2-4670-b801-0acce83c6faf","Type":"ContainerStarted","Data":"acc68beec13eaaba9fecc2251f46b457e4768bd8cf8d79a1cbae0ed0419a14a1"}
Sep 17 17:42:18.767863 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:18.767850    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8dcb3c07-9ad2-4670-b801-0acce83c6faf","Type":"ContainerStarted","Data":"050a41ae9329c65c8735fa0b973fafba72367f83bd9dd4a8d1ffb6f4575f0830"}
Sep 17 17:42:18.767992 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:18.767887    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8dcb3c07-9ad2-4670-b801-0acce83c6faf","Type":"ContainerStarted","Data":"273064e4bedf2b1324798b8f2aafaaae9ccb03717d71bb2bda26b9c63085e306"}
Sep 17 17:42:20.750605 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:20.750572    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-564fd7cf7d-hwkkx"
Sep 17 17:42:20.782343 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:20.782294    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=6.339191138 podStartE2EDuration="10.782274883s" totalImagesPullingTime="4.420590638s" totalInitContainerRuntime="22.493107ms" isStatefulPod=false podCreationTimestamp="2026-09-17 17:42:10 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:42:18.815099465 +0000 UTC m=+79.838704626" watchObservedRunningTime="2026-09-17 17:42:20.782274883 +0000 UTC m=+81.805880020"
Sep 17 17:42:20.930637 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:20.930599    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:42:21.362408 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:21.362376    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-74f79d49bf-fjjc5"
Sep 17 17:42:23.764404 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:23.764368    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 17:42:23.834077 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:23.834043    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 17:42:26.298225 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:26.298189    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-c8d8d9b9c-h5mfl"
Sep 17 17:42:26.298638 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:26.298246    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-c8d8d9b9c-h5mfl"
Sep 17 17:42:26.303122 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:26.303098    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-c8d8d9b9c-h5mfl"
Sep 17 17:42:26.802572 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:26.802546    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-c8d8d9b9c-h5mfl"
Sep 17 17:42:26.874524 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:26.874493    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5fdd747cc7-gw5hv"]
Sep 17 17:42:27.037243 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:27.037212    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 17:42:27.142958 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:27.142879    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 17:42:31.469691 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:31.469652    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 17:42:31.570039 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:31.569982    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 17:42:31.672098 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:31.672064    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 17:42:32.149480 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:32.149450    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 17:42:32.250619 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:32.250561    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 17:42:36.040897 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:36.040832    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-74b7bc945c-5f8mh" podUID="338c12c5-2bd0-444a-8fb6-276caf969978" containerName="console" containerID="cri-o://efa07b0506a0817e95a3189b4ff0ff799d09f64e8cef19ffc9ee9cad3aa234c3" gracePeriod=15
Sep 17 17:42:36.275612 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:36.275590    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-74b7bc945c-5f8mh"
Sep 17 17:42:36.366596 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:36.366505    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gjdqf\" (UniqueName: \"kubernetes.io/projected/338c12c5-2bd0-444a-8fb6-276caf969978-kube-api-access-gjdqf\") pod \"338c12c5-2bd0-444a-8fb6-276caf969978\" (UID: \"338c12c5-2bd0-444a-8fb6-276caf969978\") "
Sep 17 17:42:36.366596 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:36.366567    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/338c12c5-2bd0-444a-8fb6-276caf969978-console-config\") pod \"338c12c5-2bd0-444a-8fb6-276caf969978\" (UID: \"338c12c5-2bd0-444a-8fb6-276caf969978\") "
Sep 17 17:42:36.366815 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:36.366669    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/338c12c5-2bd0-444a-8fb6-276caf969978-oauth-serving-cert\") pod \"338c12c5-2bd0-444a-8fb6-276caf969978\" (UID: \"338c12c5-2bd0-444a-8fb6-276caf969978\") "
Sep 17 17:42:36.366815 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:36.366713    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/338c12c5-2bd0-444a-8fb6-276caf969978-trusted-ca-bundle\") pod \"338c12c5-2bd0-444a-8fb6-276caf969978\" (UID: \"338c12c5-2bd0-444a-8fb6-276caf969978\") "
Sep 17 17:42:36.366815 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:36.366762    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/338c12c5-2bd0-444a-8fb6-276caf969978-console-serving-cert\") pod \"338c12c5-2bd0-444a-8fb6-276caf969978\" (UID: \"338c12c5-2bd0-444a-8fb6-276caf969978\") "
Sep 17 17:42:36.366815 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:36.366788    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/338c12c5-2bd0-444a-8fb6-276caf969978-console-oauth-config\") pod \"338c12c5-2bd0-444a-8fb6-276caf969978\" (UID: \"338c12c5-2bd0-444a-8fb6-276caf969978\") "
Sep 17 17:42:36.366815 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:36.366813    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/338c12c5-2bd0-444a-8fb6-276caf969978-service-ca\") pod \"338c12c5-2bd0-444a-8fb6-276caf969978\" (UID: \"338c12c5-2bd0-444a-8fb6-276caf969978\") "
Sep 17 17:42:36.367065 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:36.366971    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/338c12c5-2bd0-444a-8fb6-276caf969978-console-config" (OuterVolumeSpecName: "console-config") pod "338c12c5-2bd0-444a-8fb6-276caf969978" (UID: "338c12c5-2bd0-444a-8fb6-276caf969978"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:42:36.367115 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:36.367056    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/338c12c5-2bd0-444a-8fb6-276caf969978-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "338c12c5-2bd0-444a-8fb6-276caf969978" (UID: "338c12c5-2bd0-444a-8fb6-276caf969978"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:42:36.367115 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:36.367092    2567 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/338c12c5-2bd0-444a-8fb6-276caf969978-console-config\") on node \"ip-10-0-139-237.ec2.internal\" DevicePath \"\""
Sep 17 17:42:36.367246 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:36.367222    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/338c12c5-2bd0-444a-8fb6-276caf969978-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "338c12c5-2bd0-444a-8fb6-276caf969978" (UID: "338c12c5-2bd0-444a-8fb6-276caf969978"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:42:36.367331 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:36.367310    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/338c12c5-2bd0-444a-8fb6-276caf969978-service-ca" (OuterVolumeSpecName: "service-ca") pod "338c12c5-2bd0-444a-8fb6-276caf969978" (UID: "338c12c5-2bd0-444a-8fb6-276caf969978"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:42:36.368973 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:36.368940    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/338c12c5-2bd0-444a-8fb6-276caf969978-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "338c12c5-2bd0-444a-8fb6-276caf969978" (UID: "338c12c5-2bd0-444a-8fb6-276caf969978"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:42:36.369094 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:36.369012    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/338c12c5-2bd0-444a-8fb6-276caf969978-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "338c12c5-2bd0-444a-8fb6-276caf969978" (UID: "338c12c5-2bd0-444a-8fb6-276caf969978"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:42:36.369094 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:36.369058    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/338c12c5-2bd0-444a-8fb6-276caf969978-kube-api-access-gjdqf" (OuterVolumeSpecName: "kube-api-access-gjdqf") pod "338c12c5-2bd0-444a-8fb6-276caf969978" (UID: "338c12c5-2bd0-444a-8fb6-276caf969978"). InnerVolumeSpecName "kube-api-access-gjdqf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:42:36.468033 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:36.467994    2567 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/338c12c5-2bd0-444a-8fb6-276caf969978-oauth-serving-cert\") on node \"ip-10-0-139-237.ec2.internal\" DevicePath \"\""
Sep 17 17:42:36.468033 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:36.468028    2567 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/338c12c5-2bd0-444a-8fb6-276caf969978-trusted-ca-bundle\") on node \"ip-10-0-139-237.ec2.internal\" DevicePath \"\""
Sep 17 17:42:36.468033 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:36.468039    2567 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/338c12c5-2bd0-444a-8fb6-276caf969978-console-serving-cert\") on node \"ip-10-0-139-237.ec2.internal\" DevicePath \"\""
Sep 17 17:42:36.468033 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:36.468047    2567 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/338c12c5-2bd0-444a-8fb6-276caf969978-console-oauth-config\") on node \"ip-10-0-139-237.ec2.internal\" DevicePath \"\""
Sep 17 17:42:36.468261 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:36.468056    2567 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/338c12c5-2bd0-444a-8fb6-276caf969978-service-ca\") on node \"ip-10-0-139-237.ec2.internal\" DevicePath \"\""
Sep 17 17:42:36.468261 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:36.468064    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gjdqf\" (UniqueName: \"kubernetes.io/projected/338c12c5-2bd0-444a-8fb6-276caf969978-kube-api-access-gjdqf\") on node \"ip-10-0-139-237.ec2.internal\" DevicePath \"\""
Sep 17 17:42:36.839278 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:36.839247    2567 generic.go:356] "Generic (PLEG): container finished" podID="338c12c5-2bd0-444a-8fb6-276caf969978" containerID="efa07b0506a0817e95a3189b4ff0ff799d09f64e8cef19ffc9ee9cad3aa234c3" exitCode=0
Sep 17 17:42:36.839450 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:36.839296    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-74b7bc945c-5f8mh"
Sep 17 17:42:36.839450 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:36.839334    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-74b7bc945c-5f8mh" event={"ID":"338c12c5-2bd0-444a-8fb6-276caf969978","Type":"ContainerDied","Data":"efa07b0506a0817e95a3189b4ff0ff799d09f64e8cef19ffc9ee9cad3aa234c3"}
Sep 17 17:42:36.839450 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:36.839385    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-74b7bc945c-5f8mh" event={"ID":"338c12c5-2bd0-444a-8fb6-276caf969978","Type":"ContainerDied","Data":"ae558472090125cb4df039be0089ac4f4e9e6b50682944346febc72f41fb9014"}
Sep 17 17:42:36.839450 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:36.839406    2567 scope.go:117] "RemoveContainer" containerID="efa07b0506a0817e95a3189b4ff0ff799d09f64e8cef19ffc9ee9cad3aa234c3"
Sep 17 17:42:36.848179 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:36.848162    2567 scope.go:117] "RemoveContainer" containerID="efa07b0506a0817e95a3189b4ff0ff799d09f64e8cef19ffc9ee9cad3aa234c3"
Sep 17 17:42:36.848444 ip-10-0-139-237 kubenswrapper[2567]: E0917 17:42:36.848423    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"efa07b0506a0817e95a3189b4ff0ff799d09f64e8cef19ffc9ee9cad3aa234c3\": container with ID starting with efa07b0506a0817e95a3189b4ff0ff799d09f64e8cef19ffc9ee9cad3aa234c3 not found: ID does not exist" containerID="efa07b0506a0817e95a3189b4ff0ff799d09f64e8cef19ffc9ee9cad3aa234c3"
Sep 17 17:42:36.848495 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:36.848452    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"efa07b0506a0817e95a3189b4ff0ff799d09f64e8cef19ffc9ee9cad3aa234c3"} err="failed to get container status \"efa07b0506a0817e95a3189b4ff0ff799d09f64e8cef19ffc9ee9cad3aa234c3\": rpc error: code = NotFound desc = could not find container \"efa07b0506a0817e95a3189b4ff0ff799d09f64e8cef19ffc9ee9cad3aa234c3\": container with ID starting with efa07b0506a0817e95a3189b4ff0ff799d09f64e8cef19ffc9ee9cad3aa234c3 not found: ID does not exist"
Sep 17 17:42:36.866261 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:36.866234    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-74b7bc945c-5f8mh"]
Sep 17 17:42:36.870091 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:36.870069    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-74b7bc945c-5f8mh"]
Sep 17 17:42:37.286423 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:37.286351    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="338c12c5-2bd0-444a-8fb6-276caf969978" path="/var/lib/kubelet/pods/338c12c5-2bd0-444a-8fb6-276caf969978/volumes"
Sep 17 17:42:40.965131 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:40.965060    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-74f79d49bf-fjjc5" podUID="1c94b604-5a62-4fff-a6d4-a21c31c62da0" containerName="console" containerID="cri-o://3057c7ce83e1789950d56af28b4c261f7a844b21457a9b0cf453e8b86136d040" gracePeriod=15
Sep 17 17:42:41.233956 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:41.233890    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-74f79d49bf-fjjc5"
Sep 17 17:42:41.309825 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:41.309798    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bt4k2\" (UniqueName: \"kubernetes.io/projected/1c94b604-5a62-4fff-a6d4-a21c31c62da0-kube-api-access-bt4k2\") pod \"1c94b604-5a62-4fff-a6d4-a21c31c62da0\" (UID: \"1c94b604-5a62-4fff-a6d4-a21c31c62da0\") "
Sep 17 17:42:41.309825 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:41.309831    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1c94b604-5a62-4fff-a6d4-a21c31c62da0-trusted-ca-bundle\") pod \"1c94b604-5a62-4fff-a6d4-a21c31c62da0\" (UID: \"1c94b604-5a62-4fff-a6d4-a21c31c62da0\") "
Sep 17 17:42:41.310042 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:41.309847    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1c94b604-5a62-4fff-a6d4-a21c31c62da0-oauth-serving-cert\") pod \"1c94b604-5a62-4fff-a6d4-a21c31c62da0\" (UID: \"1c94b604-5a62-4fff-a6d4-a21c31c62da0\") "
Sep 17 17:42:41.310042 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:41.309867    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1c94b604-5a62-4fff-a6d4-a21c31c62da0-service-ca\") pod \"1c94b604-5a62-4fff-a6d4-a21c31c62da0\" (UID: \"1c94b604-5a62-4fff-a6d4-a21c31c62da0\") "
Sep 17 17:42:41.310042 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:41.309902    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1c94b604-5a62-4fff-a6d4-a21c31c62da0-console-serving-cert\") pod \"1c94b604-5a62-4fff-a6d4-a21c31c62da0\" (UID: \"1c94b604-5a62-4fff-a6d4-a21c31c62da0\") "
Sep 17 17:42:41.310042 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:41.309931    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1c94b604-5a62-4fff-a6d4-a21c31c62da0-console-oauth-config\") pod \"1c94b604-5a62-4fff-a6d4-a21c31c62da0\" (UID: \"1c94b604-5a62-4fff-a6d4-a21c31c62da0\") "
Sep 17 17:42:41.310042 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:41.309966    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1c94b604-5a62-4fff-a6d4-a21c31c62da0-console-config\") pod \"1c94b604-5a62-4fff-a6d4-a21c31c62da0\" (UID: \"1c94b604-5a62-4fff-a6d4-a21c31c62da0\") "
Sep 17 17:42:41.310278 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:41.310237    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c94b604-5a62-4fff-a6d4-a21c31c62da0-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "1c94b604-5a62-4fff-a6d4-a21c31c62da0" (UID: "1c94b604-5a62-4fff-a6d4-a21c31c62da0"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:42:41.310410 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:41.310384    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c94b604-5a62-4fff-a6d4-a21c31c62da0-service-ca" (OuterVolumeSpecName: "service-ca") pod "1c94b604-5a62-4fff-a6d4-a21c31c62da0" (UID: "1c94b604-5a62-4fff-a6d4-a21c31c62da0"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:42:41.310558 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:41.310427    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c94b604-5a62-4fff-a6d4-a21c31c62da0-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "1c94b604-5a62-4fff-a6d4-a21c31c62da0" (UID: "1c94b604-5a62-4fff-a6d4-a21c31c62da0"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:42:41.310558 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:41.310465    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c94b604-5a62-4fff-a6d4-a21c31c62da0-console-config" (OuterVolumeSpecName: "console-config") pod "1c94b604-5a62-4fff-a6d4-a21c31c62da0" (UID: "1c94b604-5a62-4fff-a6d4-a21c31c62da0"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:42:41.312176 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:41.312146    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1c94b604-5a62-4fff-a6d4-a21c31c62da0-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "1c94b604-5a62-4fff-a6d4-a21c31c62da0" (UID: "1c94b604-5a62-4fff-a6d4-a21c31c62da0"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:42:41.312277 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:41.312184    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c94b604-5a62-4fff-a6d4-a21c31c62da0-kube-api-access-bt4k2" (OuterVolumeSpecName: "kube-api-access-bt4k2") pod "1c94b604-5a62-4fff-a6d4-a21c31c62da0" (UID: "1c94b604-5a62-4fff-a6d4-a21c31c62da0"). InnerVolumeSpecName "kube-api-access-bt4k2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:42:41.312319 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:41.312275    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1c94b604-5a62-4fff-a6d4-a21c31c62da0-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "1c94b604-5a62-4fff-a6d4-a21c31c62da0" (UID: "1c94b604-5a62-4fff-a6d4-a21c31c62da0"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:42:41.410802 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:41.410772    2567 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1c94b604-5a62-4fff-a6d4-a21c31c62da0-service-ca\") on node \"ip-10-0-139-237.ec2.internal\" DevicePath \"\""
Sep 17 17:42:41.410802 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:41.410796    2567 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1c94b604-5a62-4fff-a6d4-a21c31c62da0-console-serving-cert\") on node \"ip-10-0-139-237.ec2.internal\" DevicePath \"\""
Sep 17 17:42:41.410802 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:41.410806    2567 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1c94b604-5a62-4fff-a6d4-a21c31c62da0-console-oauth-config\") on node \"ip-10-0-139-237.ec2.internal\" DevicePath \"\""
Sep 17 17:42:41.410987 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:41.410816    2567 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1c94b604-5a62-4fff-a6d4-a21c31c62da0-console-config\") on node \"ip-10-0-139-237.ec2.internal\" DevicePath \"\""
Sep 17 17:42:41.410987 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:41.410825    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bt4k2\" (UniqueName: \"kubernetes.io/projected/1c94b604-5a62-4fff-a6d4-a21c31c62da0-kube-api-access-bt4k2\") on node \"ip-10-0-139-237.ec2.internal\" DevicePath \"\""
Sep 17 17:42:41.410987 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:41.410833    2567 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1c94b604-5a62-4fff-a6d4-a21c31c62da0-trusted-ca-bundle\") on node \"ip-10-0-139-237.ec2.internal\" DevicePath \"\""
Sep 17 17:42:41.410987 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:41.410840    2567 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1c94b604-5a62-4fff-a6d4-a21c31c62da0-oauth-serving-cert\") on node \"ip-10-0-139-237.ec2.internal\" DevicePath \"\""
Sep 17 17:42:41.859388 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:41.859351    2567 generic.go:356] "Generic (PLEG): container finished" podID="1c94b604-5a62-4fff-a6d4-a21c31c62da0" containerID="3057c7ce83e1789950d56af28b4c261f7a844b21457a9b0cf453e8b86136d040" exitCode=0
Sep 17 17:42:41.859605 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:41.859404    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-74f79d49bf-fjjc5"
Sep 17 17:42:41.859605 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:41.859410    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-74f79d49bf-fjjc5" event={"ID":"1c94b604-5a62-4fff-a6d4-a21c31c62da0","Type":"ContainerDied","Data":"3057c7ce83e1789950d56af28b4c261f7a844b21457a9b0cf453e8b86136d040"}
Sep 17 17:42:41.859605 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:41.859516    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-74f79d49bf-fjjc5" event={"ID":"1c94b604-5a62-4fff-a6d4-a21c31c62da0","Type":"ContainerDied","Data":"b498c6797110612bfccd4c8c90e106e47708d516879fa0181b96b211525f3199"}
Sep 17 17:42:41.859605 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:41.859589    2567 scope.go:117] "RemoveContainer" containerID="3057c7ce83e1789950d56af28b4c261f7a844b21457a9b0cf453e8b86136d040"
Sep 17 17:42:41.867801 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:41.867787    2567 scope.go:117] "RemoveContainer" containerID="3057c7ce83e1789950d56af28b4c261f7a844b21457a9b0cf453e8b86136d040"
Sep 17 17:42:41.868066 ip-10-0-139-237 kubenswrapper[2567]: E0917 17:42:41.868046    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3057c7ce83e1789950d56af28b4c261f7a844b21457a9b0cf453e8b86136d040\": container with ID starting with 3057c7ce83e1789950d56af28b4c261f7a844b21457a9b0cf453e8b86136d040 not found: ID does not exist" containerID="3057c7ce83e1789950d56af28b4c261f7a844b21457a9b0cf453e8b86136d040"
Sep 17 17:42:41.868133 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:41.868073    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3057c7ce83e1789950d56af28b4c261f7a844b21457a9b0cf453e8b86136d040"} err="failed to get container status \"3057c7ce83e1789950d56af28b4c261f7a844b21457a9b0cf453e8b86136d040\": rpc error: code = NotFound desc = could not find container \"3057c7ce83e1789950d56af28b4c261f7a844b21457a9b0cf453e8b86136d040\": container with ID starting with 3057c7ce83e1789950d56af28b4c261f7a844b21457a9b0cf453e8b86136d040 not found: ID does not exist"
Sep 17 17:42:41.880751 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:41.880728    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-74f79d49bf-fjjc5"]
Sep 17 17:42:41.884954 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:41.884934    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-74f79d49bf-fjjc5"]
Sep 17 17:42:43.288110 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:43.288079    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1c94b604-5a62-4fff-a6d4-a21c31c62da0" path="/var/lib/kubelet/pods/1c94b604-5a62-4fff-a6d4-a21c31c62da0/volumes"
Sep 17 17:42:44.739046 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:44.739017    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-4cb4f"
Sep 17 17:42:47.168749 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:47.168711    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 17:42:47.269670 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:47.269623    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 17:42:48.897446 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:48.897398    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-84c885658c-wg927_db8f376a-5499-4162-b13f-f552de771aae/kube-state-metrics/0.log"
Sep 17 17:42:49.099583 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:49.099552    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-84c885658c-wg927_db8f376a-5499-4162-b13f-f552de771aae/kube-rbac-proxy-main/0.log"
Sep 17 17:42:49.297867 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:49.297844    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-84c885658c-wg927_db8f376a-5499-4162-b13f-f552de771aae/kube-rbac-proxy-self/0.log"
Sep 17 17:42:50.748424 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:50.748392    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 17:42:50.849293 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:50.849264    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 17:42:51.097071 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:51.097049    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-wdlzx_92415019-ec6d-46b5-99b0-0be3386db8bc/init-textfile/0.log"
Sep 17 17:42:51.297565 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:51.297525    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-wdlzx_92415019-ec6d-46b5-99b0-0be3386db8bc/node-exporter/0.log"
Sep 17 17:42:51.497001 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:51.496929    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-wdlzx_92415019-ec6d-46b5-99b0-0be3386db8bc/kube-rbac-proxy/0.log"
Sep 17 17:42:51.894207 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:51.894151    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-5fdd747cc7-gw5hv" podUID="23dbde68-e2d6-4eeb-9132-5f6c29b4d261" containerName="console" containerID="cri-o://f3220517bcf3d3ddbe319912a49a1602a3c5de5dfd98df22b770e04aa5c8f4a4" gracePeriod=15
Sep 17 17:42:52.138187 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:52.138166    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5fdd747cc7-gw5hv"
Sep 17 17:42:52.198608 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:52.198504    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/23dbde68-e2d6-4eeb-9132-5f6c29b4d261-console-serving-cert\") pod \"23dbde68-e2d6-4eeb-9132-5f6c29b4d261\" (UID: \"23dbde68-e2d6-4eeb-9132-5f6c29b4d261\") "
Sep 17 17:42:52.198608 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:52.198556    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/23dbde68-e2d6-4eeb-9132-5f6c29b4d261-console-config\") pod \"23dbde68-e2d6-4eeb-9132-5f6c29b4d261\" (UID: \"23dbde68-e2d6-4eeb-9132-5f6c29b4d261\") "
Sep 17 17:42:52.198608 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:52.198576    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/23dbde68-e2d6-4eeb-9132-5f6c29b4d261-console-oauth-config\") pod \"23dbde68-e2d6-4eeb-9132-5f6c29b4d261\" (UID: \"23dbde68-e2d6-4eeb-9132-5f6c29b4d261\") "
Sep 17 17:42:52.198855 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:52.198611    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/23dbde68-e2d6-4eeb-9132-5f6c29b4d261-oauth-serving-cert\") pod \"23dbde68-e2d6-4eeb-9132-5f6c29b4d261\" (UID: \"23dbde68-e2d6-4eeb-9132-5f6c29b4d261\") "
Sep 17 17:42:52.198855 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:52.198671    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7zcw7\" (UniqueName: \"kubernetes.io/projected/23dbde68-e2d6-4eeb-9132-5f6c29b4d261-kube-api-access-7zcw7\") pod \"23dbde68-e2d6-4eeb-9132-5f6c29b4d261\" (UID: \"23dbde68-e2d6-4eeb-9132-5f6c29b4d261\") "
Sep 17 17:42:52.198855 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:52.198737    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/23dbde68-e2d6-4eeb-9132-5f6c29b4d261-service-ca\") pod \"23dbde68-e2d6-4eeb-9132-5f6c29b4d261\" (UID: \"23dbde68-e2d6-4eeb-9132-5f6c29b4d261\") "
Sep 17 17:42:52.199025 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:52.198997    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23dbde68-e2d6-4eeb-9132-5f6c29b4d261-console-config" (OuterVolumeSpecName: "console-config") pod "23dbde68-e2d6-4eeb-9132-5f6c29b4d261" (UID: "23dbde68-e2d6-4eeb-9132-5f6c29b4d261"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:42:52.199092 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:52.199015    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23dbde68-e2d6-4eeb-9132-5f6c29b4d261-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "23dbde68-e2d6-4eeb-9132-5f6c29b4d261" (UID: "23dbde68-e2d6-4eeb-9132-5f6c29b4d261"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:42:52.199254 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:52.199213    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23dbde68-e2d6-4eeb-9132-5f6c29b4d261-service-ca" (OuterVolumeSpecName: "service-ca") pod "23dbde68-e2d6-4eeb-9132-5f6c29b4d261" (UID: "23dbde68-e2d6-4eeb-9132-5f6c29b4d261"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:42:52.200808 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:52.200782    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/23dbde68-e2d6-4eeb-9132-5f6c29b4d261-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "23dbde68-e2d6-4eeb-9132-5f6c29b4d261" (UID: "23dbde68-e2d6-4eeb-9132-5f6c29b4d261"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:42:52.200897 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:52.200837    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/23dbde68-e2d6-4eeb-9132-5f6c29b4d261-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "23dbde68-e2d6-4eeb-9132-5f6c29b4d261" (UID: "23dbde68-e2d6-4eeb-9132-5f6c29b4d261"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:42:52.200897 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:52.200838    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23dbde68-e2d6-4eeb-9132-5f6c29b4d261-kube-api-access-7zcw7" (OuterVolumeSpecName: "kube-api-access-7zcw7") pod "23dbde68-e2d6-4eeb-9132-5f6c29b4d261" (UID: "23dbde68-e2d6-4eeb-9132-5f6c29b4d261"). InnerVolumeSpecName "kube-api-access-7zcw7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:42:52.297281 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:52.297256    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_8dcb3c07-9ad2-4670-b801-0acce83c6faf/init-config-reloader/0.log"
Sep 17 17:42:52.299517 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:52.299494    2567 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/23dbde68-e2d6-4eeb-9132-5f6c29b4d261-service-ca\") on node \"ip-10-0-139-237.ec2.internal\" DevicePath \"\""
Sep 17 17:42:52.299517 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:52.299514    2567 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/23dbde68-e2d6-4eeb-9132-5f6c29b4d261-console-serving-cert\") on node \"ip-10-0-139-237.ec2.internal\" DevicePath \"\""
Sep 17 17:42:52.299668 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:52.299524    2567 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/23dbde68-e2d6-4eeb-9132-5f6c29b4d261-console-config\") on node \"ip-10-0-139-237.ec2.internal\" DevicePath \"\""
Sep 17 17:42:52.299668 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:52.299545    2567 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/23dbde68-e2d6-4eeb-9132-5f6c29b4d261-console-oauth-config\") on node \"ip-10-0-139-237.ec2.internal\" DevicePath \"\""
Sep 17 17:42:52.299668 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:52.299554    2567 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/23dbde68-e2d6-4eeb-9132-5f6c29b4d261-oauth-serving-cert\") on node \"ip-10-0-139-237.ec2.internal\" DevicePath \"\""
Sep 17 17:42:52.299668 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:52.299562    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7zcw7\" (UniqueName: \"kubernetes.io/projected/23dbde68-e2d6-4eeb-9132-5f6c29b4d261-kube-api-access-7zcw7\") on node \"ip-10-0-139-237.ec2.internal\" DevicePath \"\""
Sep 17 17:42:52.497670 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:52.497599    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_8dcb3c07-9ad2-4670-b801-0acce83c6faf/prometheus/0.log"
Sep 17 17:42:52.704547 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:52.704517    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_8dcb3c07-9ad2-4670-b801-0acce83c6faf/config-reloader/0.log"
Sep 17 17:42:52.898139 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:52.898113    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_8dcb3c07-9ad2-4670-b801-0acce83c6faf/thanos-sidecar/0.log"
Sep 17 17:42:52.903161 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:52.903137    2567 generic.go:356] "Generic (PLEG): container finished" podID="23dbde68-e2d6-4eeb-9132-5f6c29b4d261" containerID="f3220517bcf3d3ddbe319912a49a1602a3c5de5dfd98df22b770e04aa5c8f4a4" exitCode=0
Sep 17 17:42:52.903268 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:52.903181    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5fdd747cc7-gw5hv"
Sep 17 17:42:52.903268 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:52.903221    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5fdd747cc7-gw5hv" event={"ID":"23dbde68-e2d6-4eeb-9132-5f6c29b4d261","Type":"ContainerDied","Data":"f3220517bcf3d3ddbe319912a49a1602a3c5de5dfd98df22b770e04aa5c8f4a4"}
Sep 17 17:42:52.903344 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:52.903271    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5fdd747cc7-gw5hv" event={"ID":"23dbde68-e2d6-4eeb-9132-5f6c29b4d261","Type":"ContainerDied","Data":"2fb0fcd4eb23e2373359cd334d38b011218cf80153f1c4c93b16f577dfe91d65"}
Sep 17 17:42:52.903344 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:52.903294    2567 scope.go:117] "RemoveContainer" containerID="f3220517bcf3d3ddbe319912a49a1602a3c5de5dfd98df22b770e04aa5c8f4a4"
Sep 17 17:42:52.911816 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:52.911800    2567 scope.go:117] "RemoveContainer" containerID="f3220517bcf3d3ddbe319912a49a1602a3c5de5dfd98df22b770e04aa5c8f4a4"
Sep 17 17:42:52.912094 ip-10-0-139-237 kubenswrapper[2567]: E0917 17:42:52.912074    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f3220517bcf3d3ddbe319912a49a1602a3c5de5dfd98df22b770e04aa5c8f4a4\": container with ID starting with f3220517bcf3d3ddbe319912a49a1602a3c5de5dfd98df22b770e04aa5c8f4a4 not found: ID does not exist" containerID="f3220517bcf3d3ddbe319912a49a1602a3c5de5dfd98df22b770e04aa5c8f4a4"
Sep 17 17:42:52.912162 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:52.912101    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f3220517bcf3d3ddbe319912a49a1602a3c5de5dfd98df22b770e04aa5c8f4a4"} err="failed to get container status \"f3220517bcf3d3ddbe319912a49a1602a3c5de5dfd98df22b770e04aa5c8f4a4\": rpc error: code = NotFound desc = could not find container \"f3220517bcf3d3ddbe319912a49a1602a3c5de5dfd98df22b770e04aa5c8f4a4\": container with ID starting with f3220517bcf3d3ddbe319912a49a1602a3c5de5dfd98df22b770e04aa5c8f4a4 not found: ID does not exist"
Sep 17 17:42:52.924287 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:52.924267    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5fdd747cc7-gw5hv"]
Sep 17 17:42:52.928098 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:52.928078    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5fdd747cc7-gw5hv"]
Sep 17 17:42:53.101100 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:53.101075    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_8dcb3c07-9ad2-4670-b801-0acce83c6faf/kube-rbac-proxy-web/0.log"
Sep 17 17:42:53.289664 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:53.287287    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="23dbde68-e2d6-4eeb-9132-5f6c29b4d261" path="/var/lib/kubelet/pods/23dbde68-e2d6-4eeb-9132-5f6c29b4d261/volumes"
Sep 17 17:42:53.297280 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:53.297257    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_8dcb3c07-9ad2-4670-b801-0acce83c6faf/kube-rbac-proxy/0.log"
Sep 17 17:42:53.496648 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:53.496615    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_8dcb3c07-9ad2-4670-b801-0acce83c6faf/kube-rbac-proxy-thanos/0.log"
Sep 17 17:42:53.697799 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:53.697725    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-7bd5d56d6-v4vgx_14c1b35f-ef7b-463f-a407-4205465909ac/prometheus-operator/0.log"
Sep 17 17:42:53.897426 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:53.897395    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-7bd5d56d6-v4vgx_14c1b35f-ef7b-463f-a407-4205465909ac/kube-rbac-proxy/0.log"
Sep 17 17:42:54.098686 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:54.098661    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-5c6747dc7b-m66j7_026b6ea8-2b9e-4f09-97a1-dc77ef61a70c/prometheus-operator-admission-webhook/0.log"
Sep 17 17:42:54.302076 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:54.302047    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-8584f8444-pjmjp_d9aa6259-82fb-4485-8fc8-4afdf2a74d29/telemeter-client/0.log"
Sep 17 17:42:54.497726 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:54.497653    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-8584f8444-pjmjp_d9aa6259-82fb-4485-8fc8-4afdf2a74d29/reload/0.log"
Sep 17 17:42:54.698048 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:54.698017    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-8584f8444-pjmjp_d9aa6259-82fb-4485-8fc8-4afdf2a74d29/kube-rbac-proxy/0.log"
Sep 17 17:42:54.897426 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:54.897402    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-564fd7cf7d-hwkkx_38d1179d-e6c3-4949-bf01-b6ed37451005/thanos-query/0.log"
Sep 17 17:42:55.096372 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:55.096343    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-564fd7cf7d-hwkkx_38d1179d-e6c3-4949-bf01-b6ed37451005/kube-rbac-proxy-web/0.log"
Sep 17 17:42:55.297356 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:55.297329    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-564fd7cf7d-hwkkx_38d1179d-e6c3-4949-bf01-b6ed37451005/kube-rbac-proxy/0.log"
Sep 17 17:42:55.496902 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:55.496873    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-564fd7cf7d-hwkkx_38d1179d-e6c3-4949-bf01-b6ed37451005/prom-label-proxy/0.log"
Sep 17 17:42:55.696788 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:55.696712    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-564fd7cf7d-hwkkx_38d1179d-e6c3-4949-bf01-b6ed37451005/kube-rbac-proxy-rules/0.log"
Sep 17 17:42:55.896488 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:55.896462    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-564fd7cf7d-hwkkx_38d1179d-e6c3-4949-bf01-b6ed37451005/kube-rbac-proxy-metrics/0.log"
Sep 17 17:42:56.899024 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:42:56.898996    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-c8d8d9b9c-h5mfl_3334a6ee-8f98-4a98-9351-1549e87d813d/console/0.log"
Sep 17 17:43:07.938735 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:07.938685    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 17:43:10.930594 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:10.930557    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:43:10.949653 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:10.949615    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:43:10.987491 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:10.987440    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:43:11.660259 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:11.660230    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 17:43:11.760561 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:11.760515    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 17:43:17.170090 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:17.170049    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 17:43:17.271055 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:17.271021    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 17:43:20.466405 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:20.466366    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 17:43:20.534417 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:20.534390    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 17:43:27.482352 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:27.482321    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7f7c6f4457-2gvv4"]
Sep 17 17:43:27.482740 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:27.482616    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1c94b604-5a62-4fff-a6d4-a21c31c62da0" containerName="console"
Sep 17 17:43:27.482740 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:27.482627    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="1c94b604-5a62-4fff-a6d4-a21c31c62da0" containerName="console"
Sep 17 17:43:27.482740 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:27.482643    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="23dbde68-e2d6-4eeb-9132-5f6c29b4d261" containerName="console"
Sep 17 17:43:27.482740 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:27.482648    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="23dbde68-e2d6-4eeb-9132-5f6c29b4d261" containerName="console"
Sep 17 17:43:27.482740 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:27.482659    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="338c12c5-2bd0-444a-8fb6-276caf969978" containerName="console"
Sep 17 17:43:27.482740 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:27.482665    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="338c12c5-2bd0-444a-8fb6-276caf969978" containerName="console"
Sep 17 17:43:27.482740 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:27.482724    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="338c12c5-2bd0-444a-8fb6-276caf969978" containerName="console"
Sep 17 17:43:27.482740 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:27.482734    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="23dbde68-e2d6-4eeb-9132-5f6c29b4d261" containerName="console"
Sep 17 17:43:27.482740 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:27.482739    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="1c94b604-5a62-4fff-a6d4-a21c31c62da0" containerName="console"
Sep 17 17:43:27.485179 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:27.485163    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7f7c6f4457-2gvv4"
Sep 17 17:43:27.501430 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:27.501405    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7f7c6f4457-2gvv4"]
Sep 17 17:43:27.545991 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:27.545963    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 17:43:27.598510 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:27.598484    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d1d7e2c1-4706-45de-87fa-7d8d7c7ddfe4-console-oauth-config\") pod \"console-7f7c6f4457-2gvv4\" (UID: \"d1d7e2c1-4706-45de-87fa-7d8d7c7ddfe4\") " pod="openshift-console/console-7f7c6f4457-2gvv4"
Sep 17 17:43:27.598653 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:27.598521    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d1d7e2c1-4706-45de-87fa-7d8d7c7ddfe4-console-serving-cert\") pod \"console-7f7c6f4457-2gvv4\" (UID: \"d1d7e2c1-4706-45de-87fa-7d8d7c7ddfe4\") " pod="openshift-console/console-7f7c6f4457-2gvv4"
Sep 17 17:43:27.598653 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:27.598565    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d1d7e2c1-4706-45de-87fa-7d8d7c7ddfe4-trusted-ca-bundle\") pod \"console-7f7c6f4457-2gvv4\" (UID: \"d1d7e2c1-4706-45de-87fa-7d8d7c7ddfe4\") " pod="openshift-console/console-7f7c6f4457-2gvv4"
Sep 17 17:43:27.598653 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:27.598599    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d1d7e2c1-4706-45de-87fa-7d8d7c7ddfe4-oauth-serving-cert\") pod \"console-7f7c6f4457-2gvv4\" (UID: \"d1d7e2c1-4706-45de-87fa-7d8d7c7ddfe4\") " pod="openshift-console/console-7f7c6f4457-2gvv4"
Sep 17 17:43:27.598749 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:27.598679    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bn24n\" (UniqueName: \"kubernetes.io/projected/d1d7e2c1-4706-45de-87fa-7d8d7c7ddfe4-kube-api-access-bn24n\") pod \"console-7f7c6f4457-2gvv4\" (UID: \"d1d7e2c1-4706-45de-87fa-7d8d7c7ddfe4\") " pod="openshift-console/console-7f7c6f4457-2gvv4"
Sep 17 17:43:27.598749 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:27.598708    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d1d7e2c1-4706-45de-87fa-7d8d7c7ddfe4-console-config\") pod \"console-7f7c6f4457-2gvv4\" (UID: \"d1d7e2c1-4706-45de-87fa-7d8d7c7ddfe4\") " pod="openshift-console/console-7f7c6f4457-2gvv4"
Sep 17 17:43:27.598812 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:27.598747    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d1d7e2c1-4706-45de-87fa-7d8d7c7ddfe4-service-ca\") pod \"console-7f7c6f4457-2gvv4\" (UID: \"d1d7e2c1-4706-45de-87fa-7d8d7c7ddfe4\") " pod="openshift-console/console-7f7c6f4457-2gvv4"
Sep 17 17:43:27.640716 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:27.640687    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 17:43:27.699050 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:27.699024    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 17:43:27.699194 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:27.699176    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d1d7e2c1-4706-45de-87fa-7d8d7c7ddfe4-trusted-ca-bundle\") pod \"console-7f7c6f4457-2gvv4\" (UID: \"d1d7e2c1-4706-45de-87fa-7d8d7c7ddfe4\") " pod="openshift-console/console-7f7c6f4457-2gvv4"
Sep 17 17:43:27.699239 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:27.699204    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d1d7e2c1-4706-45de-87fa-7d8d7c7ddfe4-oauth-serving-cert\") pod \"console-7f7c6f4457-2gvv4\" (UID: \"d1d7e2c1-4706-45de-87fa-7d8d7c7ddfe4\") " pod="openshift-console/console-7f7c6f4457-2gvv4"
Sep 17 17:43:27.699426 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:27.699391    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bn24n\" (UniqueName: \"kubernetes.io/projected/d1d7e2c1-4706-45de-87fa-7d8d7c7ddfe4-kube-api-access-bn24n\") pod \"console-7f7c6f4457-2gvv4\" (UID: \"d1d7e2c1-4706-45de-87fa-7d8d7c7ddfe4\") " pod="openshift-console/console-7f7c6f4457-2gvv4"
Sep 17 17:43:27.699576 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:27.699454    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d1d7e2c1-4706-45de-87fa-7d8d7c7ddfe4-console-config\") pod \"console-7f7c6f4457-2gvv4\" (UID: \"d1d7e2c1-4706-45de-87fa-7d8d7c7ddfe4\") " pod="openshift-console/console-7f7c6f4457-2gvv4"
Sep 17 17:43:27.699576 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:27.699507    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d1d7e2c1-4706-45de-87fa-7d8d7c7ddfe4-service-ca\") pod \"console-7f7c6f4457-2gvv4\" (UID: \"d1d7e2c1-4706-45de-87fa-7d8d7c7ddfe4\") " pod="openshift-console/console-7f7c6f4457-2gvv4"
Sep 17 17:43:27.699576 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:27.699573    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d1d7e2c1-4706-45de-87fa-7d8d7c7ddfe4-console-oauth-config\") pod \"console-7f7c6f4457-2gvv4\" (UID: \"d1d7e2c1-4706-45de-87fa-7d8d7c7ddfe4\") " pod="openshift-console/console-7f7c6f4457-2gvv4"
Sep 17 17:43:27.699734 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:27.699612    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d1d7e2c1-4706-45de-87fa-7d8d7c7ddfe4-console-serving-cert\") pod \"console-7f7c6f4457-2gvv4\" (UID: \"d1d7e2c1-4706-45de-87fa-7d8d7c7ddfe4\") " pod="openshift-console/console-7f7c6f4457-2gvv4"
Sep 17 17:43:27.700042 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:27.700015    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d1d7e2c1-4706-45de-87fa-7d8d7c7ddfe4-oauth-serving-cert\") pod \"console-7f7c6f4457-2gvv4\" (UID: \"d1d7e2c1-4706-45de-87fa-7d8d7c7ddfe4\") " pod="openshift-console/console-7f7c6f4457-2gvv4"
Sep 17 17:43:27.700129 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:27.700083    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d1d7e2c1-4706-45de-87fa-7d8d7c7ddfe4-trusted-ca-bundle\") pod \"console-7f7c6f4457-2gvv4\" (UID: \"d1d7e2c1-4706-45de-87fa-7d8d7c7ddfe4\") " pod="openshift-console/console-7f7c6f4457-2gvv4"
Sep 17 17:43:27.700429 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:27.700412    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d1d7e2c1-4706-45de-87fa-7d8d7c7ddfe4-console-config\") pod \"console-7f7c6f4457-2gvv4\" (UID: \"d1d7e2c1-4706-45de-87fa-7d8d7c7ddfe4\") " pod="openshift-console/console-7f7c6f4457-2gvv4"
Sep 17 17:43:27.700499 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:27.700475    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d1d7e2c1-4706-45de-87fa-7d8d7c7ddfe4-service-ca\") pod \"console-7f7c6f4457-2gvv4\" (UID: \"d1d7e2c1-4706-45de-87fa-7d8d7c7ddfe4\") " pod="openshift-console/console-7f7c6f4457-2gvv4"
Sep 17 17:43:27.702119 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:27.702094    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d1d7e2c1-4706-45de-87fa-7d8d7c7ddfe4-console-serving-cert\") pod \"console-7f7c6f4457-2gvv4\" (UID: \"d1d7e2c1-4706-45de-87fa-7d8d7c7ddfe4\") " pod="openshift-console/console-7f7c6f4457-2gvv4"
Sep 17 17:43:27.702200 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:27.702146    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d1d7e2c1-4706-45de-87fa-7d8d7c7ddfe4-console-oauth-config\") pod \"console-7f7c6f4457-2gvv4\" (UID: \"d1d7e2c1-4706-45de-87fa-7d8d7c7ddfe4\") " pod="openshift-console/console-7f7c6f4457-2gvv4"
Sep 17 17:43:27.708736 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:27.708712    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bn24n\" (UniqueName: \"kubernetes.io/projected/d1d7e2c1-4706-45de-87fa-7d8d7c7ddfe4-kube-api-access-bn24n\") pod \"console-7f7c6f4457-2gvv4\" (UID: \"d1d7e2c1-4706-45de-87fa-7d8d7c7ddfe4\") " pod="openshift-console/console-7f7c6f4457-2gvv4"
Sep 17 17:43:27.793898 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:27.793877    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7f7c6f4457-2gvv4"
Sep 17 17:43:28.064799 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:28.064776    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7f7c6f4457-2gvv4"]
Sep 17 17:43:28.066852 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:43:28.066824    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1d7e2c1_4706_45de_87fa_7d8d7c7ddfe4.slice/crio-195a47adf75b77d14e5c8f26fc660c4eb46c2bf6de0c6ed8c5299fae202359ad WatchSource:0}: Error finding container 195a47adf75b77d14e5c8f26fc660c4eb46c2bf6de0c6ed8c5299fae202359ad: Status 404 returned error can't find the container with id 195a47adf75b77d14e5c8f26fc660c4eb46c2bf6de0c6ed8c5299fae202359ad
Sep 17 17:43:28.274857 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:28.274814    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 17:43:28.375287 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:28.375190    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 17:43:28.985622 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:28.985585    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 17 17:43:28.986061 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:28.985966    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="8dcb3c07-9ad2-4670-b801-0acce83c6faf" containerName="prometheus" containerID="cri-o://273064e4bedf2b1324798b8f2aafaaae9ccb03717d71bb2bda26b9c63085e306" gracePeriod=600
Sep 17 17:43:28.986061 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:28.986002    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="8dcb3c07-9ad2-4670-b801-0acce83c6faf" containerName="kube-rbac-proxy-thanos" containerID="cri-o://d9543b6bf84dd3ea06ca00cc9480684d94b52827b927f3e39c5279872c7fbae1" gracePeriod=600
Sep 17 17:43:28.986061 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:28.986032    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="8dcb3c07-9ad2-4670-b801-0acce83c6faf" containerName="thanos-sidecar" containerID="cri-o://acc68beec13eaaba9fecc2251f46b457e4768bd8cf8d79a1cbae0ed0419a14a1" gracePeriod=600
Sep 17 17:43:28.986304 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:28.986006    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="8dcb3c07-9ad2-4670-b801-0acce83c6faf" containerName="kube-rbac-proxy" containerID="cri-o://8c82f26264ba90782e9fb15f6fbd78bdb798f51c5a2391efa2e98a5f4fe00e43" gracePeriod=600
Sep 17 17:43:28.986304 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:28.986002    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="8dcb3c07-9ad2-4670-b801-0acce83c6faf" containerName="kube-rbac-proxy-web" containerID="cri-o://20126bf6b5656fe616b70ec294e7d7a76c3f6012919f3b40668a23c78ea35d40" gracePeriod=600
Sep 17 17:43:28.986304 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:28.986033    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="8dcb3c07-9ad2-4670-b801-0acce83c6faf" containerName="config-reloader" containerID="cri-o://050a41ae9329c65c8735fa0b973fafba72367f83bd9dd4a8d1ffb6f4575f0830" gracePeriod=600
Sep 17 17:43:29.049024 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:29.048632    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7f7c6f4457-2gvv4" event={"ID":"d1d7e2c1-4706-45de-87fa-7d8d7c7ddfe4","Type":"ContainerStarted","Data":"12e3cfe49742d0ee1bf664654b4a5b44f4f5935f2516f3e1177af2acd859aa11"}
Sep 17 17:43:29.049024 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:29.048697    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7f7c6f4457-2gvv4" event={"ID":"d1d7e2c1-4706-45de-87fa-7d8d7c7ddfe4","Type":"ContainerStarted","Data":"195a47adf75b77d14e5c8f26fc660c4eb46c2bf6de0c6ed8c5299fae202359ad"}
Sep 17 17:43:29.069953 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:29.069907    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7f7c6f4457-2gvv4" podStartSLOduration=2.069888304 podStartE2EDuration="2.069888304s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:43:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:43:29.068126035 +0000 UTC m=+150.091731198" watchObservedRunningTime="2026-09-17 17:43:29.069888304 +0000 UTC m=+150.093493439"
Sep 17 17:43:29.225927 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:29.225902    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:43:29.313790 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:29.313755    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8dcb3c07-9ad2-4670-b801-0acce83c6faf-tls-assets\") pod \"8dcb3c07-9ad2-4670-b801-0acce83c6faf\" (UID: \"8dcb3c07-9ad2-4670-b801-0acce83c6faf\") "
Sep 17 17:43:29.313790 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:29.313790    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/8dcb3c07-9ad2-4670-b801-0acce83c6faf-prometheus-k8s-rulefiles-1\") pod \"8dcb3c07-9ad2-4670-b801-0acce83c6faf\" (UID: \"8dcb3c07-9ad2-4670-b801-0acce83c6faf\") "
Sep 17 17:43:29.314023 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:29.313819    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8dcb3c07-9ad2-4670-b801-0acce83c6faf-web-config\") pod \"8dcb3c07-9ad2-4670-b801-0acce83c6faf\" (UID: \"8dcb3c07-9ad2-4670-b801-0acce83c6faf\") "
Sep 17 17:43:29.314023 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:29.313839    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8dcb3c07-9ad2-4670-b801-0acce83c6faf-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"8dcb3c07-9ad2-4670-b801-0acce83c6faf\" (UID: \"8dcb3c07-9ad2-4670-b801-0acce83c6faf\") "
Sep 17 17:43:29.314023 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:29.313865    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/8dcb3c07-9ad2-4670-b801-0acce83c6faf-prometheus-k8s-rulefiles-2\") pod \"8dcb3c07-9ad2-4670-b801-0acce83c6faf\" (UID: \"8dcb3c07-9ad2-4670-b801-0acce83c6faf\") "
Sep 17 17:43:29.314023 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:29.313890    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/8dcb3c07-9ad2-4670-b801-0acce83c6faf-secret-metrics-client-certs\") pod \"8dcb3c07-9ad2-4670-b801-0acce83c6faf\" (UID: \"8dcb3c07-9ad2-4670-b801-0acce83c6faf\") "
Sep 17 17:43:29.314023 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:29.313947    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/8dcb3c07-9ad2-4670-b801-0acce83c6faf-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"8dcb3c07-9ad2-4670-b801-0acce83c6faf\" (UID: \"8dcb3c07-9ad2-4670-b801-0acce83c6faf\") "
Sep 17 17:43:29.314023 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:29.313974    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/8dcb3c07-9ad2-4670-b801-0acce83c6faf-prometheus-k8s-rulefiles-0\") pod \"8dcb3c07-9ad2-4670-b801-0acce83c6faf\" (UID: \"8dcb3c07-9ad2-4670-b801-0acce83c6faf\") "
Sep 17 17:43:29.314023 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:29.314004    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8dcb3c07-9ad2-4670-b801-0acce83c6faf-configmap-metrics-client-ca\") pod \"8dcb3c07-9ad2-4670-b801-0acce83c6faf\" (UID: \"8dcb3c07-9ad2-4670-b801-0acce83c6faf\") "
Sep 17 17:43:29.314023 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:29.314025    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/8dcb3c07-9ad2-4670-b801-0acce83c6faf-config\") pod \"8dcb3c07-9ad2-4670-b801-0acce83c6faf\" (UID: \"8dcb3c07-9ad2-4670-b801-0acce83c6faf\") "
Sep 17 17:43:29.314429 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:29.314050    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8dcb3c07-9ad2-4670-b801-0acce83c6faf-configmap-kubelet-serving-ca-bundle\") pod \"8dcb3c07-9ad2-4670-b801-0acce83c6faf\" (UID: \"8dcb3c07-9ad2-4670-b801-0acce83c6faf\") "
Sep 17 17:43:29.314429 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:29.314093    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8dcb3c07-9ad2-4670-b801-0acce83c6faf-secret-kube-rbac-proxy\") pod \"8dcb3c07-9ad2-4670-b801-0acce83c6faf\" (UID: \"8dcb3c07-9ad2-4670-b801-0acce83c6faf\") "
Sep 17 17:43:29.314429 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:29.314122    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8dcb3c07-9ad2-4670-b801-0acce83c6faf-config-out\") pod \"8dcb3c07-9ad2-4670-b801-0acce83c6faf\" (UID: \"8dcb3c07-9ad2-4670-b801-0acce83c6faf\") "
Sep 17 17:43:29.314429 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:29.314149    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/8dcb3c07-9ad2-4670-b801-0acce83c6faf-secret-prometheus-k8s-tls\") pod \"8dcb3c07-9ad2-4670-b801-0acce83c6faf\" (UID: \"8dcb3c07-9ad2-4670-b801-0acce83c6faf\") "
Sep 17 17:43:29.314429 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:29.314213    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8dcb3c07-9ad2-4670-b801-0acce83c6faf-prometheus-trusted-ca-bundle\") pod \"8dcb3c07-9ad2-4670-b801-0acce83c6faf\" (UID: \"8dcb3c07-9ad2-4670-b801-0acce83c6faf\") "
Sep 17 17:43:29.314429 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:29.314264    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6t448\" (UniqueName: \"kubernetes.io/projected/8dcb3c07-9ad2-4670-b801-0acce83c6faf-kube-api-access-6t448\") pod \"8dcb3c07-9ad2-4670-b801-0acce83c6faf\" (UID: \"8dcb3c07-9ad2-4670-b801-0acce83c6faf\") "
Sep 17 17:43:29.314429 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:29.314262    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8dcb3c07-9ad2-4670-b801-0acce83c6faf-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "8dcb3c07-9ad2-4670-b801-0acce83c6faf" (UID: "8dcb3c07-9ad2-4670-b801-0acce83c6faf"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:43:29.314429 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:29.314297    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8dcb3c07-9ad2-4670-b801-0acce83c6faf-configmap-serving-certs-ca-bundle\") pod \"8dcb3c07-9ad2-4670-b801-0acce83c6faf\" (UID: \"8dcb3c07-9ad2-4670-b801-0acce83c6faf\") "
Sep 17 17:43:29.314429 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:29.314339    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/8dcb3c07-9ad2-4670-b801-0acce83c6faf-thanos-prometheus-http-client-file\") pod \"8dcb3c07-9ad2-4670-b801-0acce83c6faf\" (UID: \"8dcb3c07-9ad2-4670-b801-0acce83c6faf\") "
Sep 17 17:43:29.314429 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:29.314397    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/8dcb3c07-9ad2-4670-b801-0acce83c6faf-prometheus-k8s-db\") pod \"8dcb3c07-9ad2-4670-b801-0acce83c6faf\" (UID: \"8dcb3c07-9ad2-4670-b801-0acce83c6faf\") "
Sep 17 17:43:29.314429 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:29.314432    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8dcb3c07-9ad2-4670-b801-0acce83c6faf-secret-grpc-tls\") pod \"8dcb3c07-9ad2-4670-b801-0acce83c6faf\" (UID: \"8dcb3c07-9ad2-4670-b801-0acce83c6faf\") "
Sep 17 17:43:29.315002 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:29.314447    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8dcb3c07-9ad2-4670-b801-0acce83c6faf-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "8dcb3c07-9ad2-4670-b801-0acce83c6faf" (UID: "8dcb3c07-9ad2-4670-b801-0acce83c6faf"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:43:29.315002 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:29.314632    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8dcb3c07-9ad2-4670-b801-0acce83c6faf-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "8dcb3c07-9ad2-4670-b801-0acce83c6faf" (UID: "8dcb3c07-9ad2-4670-b801-0acce83c6faf"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:43:29.315546 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:29.315499    2567 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/8dcb3c07-9ad2-4670-b801-0acce83c6faf-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-139-237.ec2.internal\" DevicePath \"\""
Sep 17 17:43:29.315721 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:29.315559    2567 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/8dcb3c07-9ad2-4670-b801-0acce83c6faf-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-139-237.ec2.internal\" DevicePath \"\""
Sep 17 17:43:29.315721 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:29.315577    2567 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8dcb3c07-9ad2-4670-b801-0acce83c6faf-configmap-metrics-client-ca\") on node \"ip-10-0-139-237.ec2.internal\" DevicePath \"\""
Sep 17 17:43:29.315927 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:29.315899    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8dcb3c07-9ad2-4670-b801-0acce83c6faf-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "8dcb3c07-9ad2-4670-b801-0acce83c6faf" (UID: "8dcb3c07-9ad2-4670-b801-0acce83c6faf"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:43:29.316070 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:29.316045    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8dcb3c07-9ad2-4670-b801-0acce83c6faf-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "8dcb3c07-9ad2-4670-b801-0acce83c6faf" (UID: "8dcb3c07-9ad2-4670-b801-0acce83c6faf"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:43:29.316669 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:29.316643    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8dcb3c07-9ad2-4670-b801-0acce83c6faf-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "8dcb3c07-9ad2-4670-b801-0acce83c6faf" (UID: "8dcb3c07-9ad2-4670-b801-0acce83c6faf"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:43:29.316938 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:29.316908    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8dcb3c07-9ad2-4670-b801-0acce83c6faf-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "8dcb3c07-9ad2-4670-b801-0acce83c6faf" (UID: "8dcb3c07-9ad2-4670-b801-0acce83c6faf"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:43:29.317465 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:29.317296    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8dcb3c07-9ad2-4670-b801-0acce83c6faf-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "8dcb3c07-9ad2-4670-b801-0acce83c6faf" (UID: "8dcb3c07-9ad2-4670-b801-0acce83c6faf"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:43:29.317828 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:29.317793    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8dcb3c07-9ad2-4670-b801-0acce83c6faf-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "8dcb3c07-9ad2-4670-b801-0acce83c6faf" (UID: "8dcb3c07-9ad2-4670-b801-0acce83c6faf"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:43:29.319327 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:29.319260    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8dcb3c07-9ad2-4670-b801-0acce83c6faf-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "8dcb3c07-9ad2-4670-b801-0acce83c6faf" (UID: "8dcb3c07-9ad2-4670-b801-0acce83c6faf"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:43:29.319509 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:29.319480    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8dcb3c07-9ad2-4670-b801-0acce83c6faf-config-out" (OuterVolumeSpecName: "config-out") pod "8dcb3c07-9ad2-4670-b801-0acce83c6faf" (UID: "8dcb3c07-9ad2-4670-b801-0acce83c6faf"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:43:29.319509 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:29.319480    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8dcb3c07-9ad2-4670-b801-0acce83c6faf-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "8dcb3c07-9ad2-4670-b801-0acce83c6faf" (UID: "8dcb3c07-9ad2-4670-b801-0acce83c6faf"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:43:29.319753 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:29.319692    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8dcb3c07-9ad2-4670-b801-0acce83c6faf-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "8dcb3c07-9ad2-4670-b801-0acce83c6faf" (UID: "8dcb3c07-9ad2-4670-b801-0acce83c6faf"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:43:29.319836 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:29.319812    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8dcb3c07-9ad2-4670-b801-0acce83c6faf-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "8dcb3c07-9ad2-4670-b801-0acce83c6faf" (UID: "8dcb3c07-9ad2-4670-b801-0acce83c6faf"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:43:29.319927 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:29.319899    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8dcb3c07-9ad2-4670-b801-0acce83c6faf-kube-api-access-6t448" (OuterVolumeSpecName: "kube-api-access-6t448") pod "8dcb3c07-9ad2-4670-b801-0acce83c6faf" (UID: "8dcb3c07-9ad2-4670-b801-0acce83c6faf"). InnerVolumeSpecName "kube-api-access-6t448". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:43:29.319927 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:29.319915    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8dcb3c07-9ad2-4670-b801-0acce83c6faf-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "8dcb3c07-9ad2-4670-b801-0acce83c6faf" (UID: "8dcb3c07-9ad2-4670-b801-0acce83c6faf"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:43:29.320042 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:29.319992    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8dcb3c07-9ad2-4670-b801-0acce83c6faf-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "8dcb3c07-9ad2-4670-b801-0acce83c6faf" (UID: "8dcb3c07-9ad2-4670-b801-0acce83c6faf"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:43:29.320097 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:29.320049    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8dcb3c07-9ad2-4670-b801-0acce83c6faf-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "8dcb3c07-9ad2-4670-b801-0acce83c6faf" (UID: "8dcb3c07-9ad2-4670-b801-0acce83c6faf"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:43:29.320585 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:29.320518    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8dcb3c07-9ad2-4670-b801-0acce83c6faf-config" (OuterVolumeSpecName: "config") pod "8dcb3c07-9ad2-4670-b801-0acce83c6faf" (UID: "8dcb3c07-9ad2-4670-b801-0acce83c6faf"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:43:29.329456 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:29.329432    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8dcb3c07-9ad2-4670-b801-0acce83c6faf-web-config" (OuterVolumeSpecName: "web-config") pod "8dcb3c07-9ad2-4670-b801-0acce83c6faf" (UID: "8dcb3c07-9ad2-4670-b801-0acce83c6faf"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:43:29.416673 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:29.416643    2567 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/8dcb3c07-9ad2-4670-b801-0acce83c6faf-prometheus-k8s-db\") on node \"ip-10-0-139-237.ec2.internal\" DevicePath \"\""
Sep 17 17:43:29.416673 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:29.416667    2567 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8dcb3c07-9ad2-4670-b801-0acce83c6faf-secret-grpc-tls\") on node \"ip-10-0-139-237.ec2.internal\" DevicePath \"\""
Sep 17 17:43:29.416673 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:29.416677    2567 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8dcb3c07-9ad2-4670-b801-0acce83c6faf-tls-assets\") on node \"ip-10-0-139-237.ec2.internal\" DevicePath \"\""
Sep 17 17:43:29.416879 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:29.416687    2567 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8dcb3c07-9ad2-4670-b801-0acce83c6faf-web-config\") on node \"ip-10-0-139-237.ec2.internal\" DevicePath \"\""
Sep 17 17:43:29.416879 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:29.416696    2567 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8dcb3c07-9ad2-4670-b801-0acce83c6faf-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-139-237.ec2.internal\" DevicePath \"\""
Sep 17 17:43:29.416879 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:29.416705    2567 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/8dcb3c07-9ad2-4670-b801-0acce83c6faf-secret-metrics-client-certs\") on node \"ip-10-0-139-237.ec2.internal\" DevicePath \"\""
Sep 17 17:43:29.416879 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:29.416713    2567 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/8dcb3c07-9ad2-4670-b801-0acce83c6faf-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-139-237.ec2.internal\" DevicePath \"\""
Sep 17 17:43:29.416879 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:29.416722    2567 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/8dcb3c07-9ad2-4670-b801-0acce83c6faf-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-139-237.ec2.internal\" DevicePath \"\""
Sep 17 17:43:29.416879 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:29.416730    2567 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/8dcb3c07-9ad2-4670-b801-0acce83c6faf-config\") on node \"ip-10-0-139-237.ec2.internal\" DevicePath \"\""
Sep 17 17:43:29.416879 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:29.416738    2567 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8dcb3c07-9ad2-4670-b801-0acce83c6faf-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-139-237.ec2.internal\" DevicePath \"\""
Sep 17 17:43:29.416879 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:29.416747    2567 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8dcb3c07-9ad2-4670-b801-0acce83c6faf-secret-kube-rbac-proxy\") on node \"ip-10-0-139-237.ec2.internal\" DevicePath \"\""
Sep 17 17:43:29.416879 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:29.416755    2567 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8dcb3c07-9ad2-4670-b801-0acce83c6faf-config-out\") on node \"ip-10-0-139-237.ec2.internal\" DevicePath \"\""
Sep 17 17:43:29.416879 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:29.416763    2567 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/8dcb3c07-9ad2-4670-b801-0acce83c6faf-secret-prometheus-k8s-tls\") on node \"ip-10-0-139-237.ec2.internal\" DevicePath \"\""
Sep 17 17:43:29.416879 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:29.416773    2567 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8dcb3c07-9ad2-4670-b801-0acce83c6faf-prometheus-trusted-ca-bundle\") on node \"ip-10-0-139-237.ec2.internal\" DevicePath \"\""
Sep 17 17:43:29.416879 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:29.416782    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6t448\" (UniqueName: \"kubernetes.io/projected/8dcb3c07-9ad2-4670-b801-0acce83c6faf-kube-api-access-6t448\") on node \"ip-10-0-139-237.ec2.internal\" DevicePath \"\""
Sep 17 17:43:29.416879 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:29.416791    2567 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8dcb3c07-9ad2-4670-b801-0acce83c6faf-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-139-237.ec2.internal\" DevicePath \"\""
Sep 17 17:43:29.416879 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:29.416799    2567 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/8dcb3c07-9ad2-4670-b801-0acce83c6faf-thanos-prometheus-http-client-file\") on node \"ip-10-0-139-237.ec2.internal\" DevicePath \"\""
Sep 17 17:43:30.021809 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:30.021712    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 17:43:30.056427 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:30.056396    2567 generic.go:356] "Generic (PLEG): container finished" podID="8dcb3c07-9ad2-4670-b801-0acce83c6faf" containerID="d9543b6bf84dd3ea06ca00cc9480684d94b52827b927f3e39c5279872c7fbae1" exitCode=0
Sep 17 17:43:30.056427 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:30.056419    2567 generic.go:356] "Generic (PLEG): container finished" podID="8dcb3c07-9ad2-4670-b801-0acce83c6faf" containerID="8c82f26264ba90782e9fb15f6fbd78bdb798f51c5a2391efa2e98a5f4fe00e43" exitCode=0
Sep 17 17:43:30.056427 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:30.056425    2567 generic.go:356] "Generic (PLEG): container finished" podID="8dcb3c07-9ad2-4670-b801-0acce83c6faf" containerID="20126bf6b5656fe616b70ec294e7d7a76c3f6012919f3b40668a23c78ea35d40" exitCode=0
Sep 17 17:43:30.056427 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:30.056431    2567 generic.go:356] "Generic (PLEG): container finished" podID="8dcb3c07-9ad2-4670-b801-0acce83c6faf" containerID="acc68beec13eaaba9fecc2251f46b457e4768bd8cf8d79a1cbae0ed0419a14a1" exitCode=0
Sep 17 17:43:30.056427 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:30.056436    2567 generic.go:356] "Generic (PLEG): container finished" podID="8dcb3c07-9ad2-4670-b801-0acce83c6faf" containerID="050a41ae9329c65c8735fa0b973fafba72367f83bd9dd4a8d1ffb6f4575f0830" exitCode=0
Sep 17 17:43:30.056720 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:30.056441    2567 generic.go:356] "Generic (PLEG): container finished" podID="8dcb3c07-9ad2-4670-b801-0acce83c6faf" containerID="273064e4bedf2b1324798b8f2aafaaae9ccb03717d71bb2bda26b9c63085e306" exitCode=0
Sep 17 17:43:30.056720 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:30.056481    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8dcb3c07-9ad2-4670-b801-0acce83c6faf","Type":"ContainerDied","Data":"d9543b6bf84dd3ea06ca00cc9480684d94b52827b927f3e39c5279872c7fbae1"}
Sep 17 17:43:30.056720 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:30.056563    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8dcb3c07-9ad2-4670-b801-0acce83c6faf","Type":"ContainerDied","Data":"8c82f26264ba90782e9fb15f6fbd78bdb798f51c5a2391efa2e98a5f4fe00e43"}
Sep 17 17:43:30.056720 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:30.056487    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:43:30.056720 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:30.056592    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8dcb3c07-9ad2-4670-b801-0acce83c6faf","Type":"ContainerDied","Data":"20126bf6b5656fe616b70ec294e7d7a76c3f6012919f3b40668a23c78ea35d40"}
Sep 17 17:43:30.056720 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:30.056619    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8dcb3c07-9ad2-4670-b801-0acce83c6faf","Type":"ContainerDied","Data":"acc68beec13eaaba9fecc2251f46b457e4768bd8cf8d79a1cbae0ed0419a14a1"}
Sep 17 17:43:30.056720 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:30.056642    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8dcb3c07-9ad2-4670-b801-0acce83c6faf","Type":"ContainerDied","Data":"050a41ae9329c65c8735fa0b973fafba72367f83bd9dd4a8d1ffb6f4575f0830"}
Sep 17 17:43:30.056720 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:30.056665    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8dcb3c07-9ad2-4670-b801-0acce83c6faf","Type":"ContainerDied","Data":"273064e4bedf2b1324798b8f2aafaaae9ccb03717d71bb2bda26b9c63085e306"}
Sep 17 17:43:30.056720 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:30.056653    2567 scope.go:117] "RemoveContainer" containerID="d9543b6bf84dd3ea06ca00cc9480684d94b52827b927f3e39c5279872c7fbae1"
Sep 17 17:43:30.056720 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:30.056688    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8dcb3c07-9ad2-4670-b801-0acce83c6faf","Type":"ContainerDied","Data":"ad2b95b28324b1cb5ff2941e177dc3e52514f505c2801fd30c1df2bea3e2b603"}
Sep 17 17:43:30.064837 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:30.064819    2567 scope.go:117] "RemoveContainer" containerID="8c82f26264ba90782e9fb15f6fbd78bdb798f51c5a2391efa2e98a5f4fe00e43"
Sep 17 17:43:30.071428 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:30.071413    2567 scope.go:117] "RemoveContainer" containerID="20126bf6b5656fe616b70ec294e7d7a76c3f6012919f3b40668a23c78ea35d40"
Sep 17 17:43:30.077563 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:30.077524    2567 scope.go:117] "RemoveContainer" containerID="acc68beec13eaaba9fecc2251f46b457e4768bd8cf8d79a1cbae0ed0419a14a1"
Sep 17 17:43:30.081993 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:30.081970    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 17 17:43:30.084363 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:30.084348    2567 scope.go:117] "RemoveContainer" containerID="050a41ae9329c65c8735fa0b973fafba72367f83bd9dd4a8d1ffb6f4575f0830"
Sep 17 17:43:30.087549 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:30.087506    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 17 17:43:30.091356 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:30.091340    2567 scope.go:117] "RemoveContainer" containerID="273064e4bedf2b1324798b8f2aafaaae9ccb03717d71bb2bda26b9c63085e306"
Sep 17 17:43:30.097842 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:30.097825    2567 scope.go:117] "RemoveContainer" containerID="3d98a78d4042469903bb7f4870086e12672504526d45866281542e69841caf04"
Sep 17 17:43:30.104087 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:30.104065    2567 scope.go:117] "RemoveContainer" containerID="d9543b6bf84dd3ea06ca00cc9480684d94b52827b927f3e39c5279872c7fbae1"
Sep 17 17:43:30.104325 ip-10-0-139-237 kubenswrapper[2567]: E0917 17:43:30.104308    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d9543b6bf84dd3ea06ca00cc9480684d94b52827b927f3e39c5279872c7fbae1\": container with ID starting with d9543b6bf84dd3ea06ca00cc9480684d94b52827b927f3e39c5279872c7fbae1 not found: ID does not exist" containerID="d9543b6bf84dd3ea06ca00cc9480684d94b52827b927f3e39c5279872c7fbae1"
Sep 17 17:43:30.104384 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:30.104332    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d9543b6bf84dd3ea06ca00cc9480684d94b52827b927f3e39c5279872c7fbae1"} err="failed to get container status \"d9543b6bf84dd3ea06ca00cc9480684d94b52827b927f3e39c5279872c7fbae1\": rpc error: code = NotFound desc = could not find container \"d9543b6bf84dd3ea06ca00cc9480684d94b52827b927f3e39c5279872c7fbae1\": container with ID starting with d9543b6bf84dd3ea06ca00cc9480684d94b52827b927f3e39c5279872c7fbae1 not found: ID does not exist"
Sep 17 17:43:30.104384 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:30.104348    2567 scope.go:117] "RemoveContainer" containerID="8c82f26264ba90782e9fb15f6fbd78bdb798f51c5a2391efa2e98a5f4fe00e43"
Sep 17 17:43:30.104610 ip-10-0-139-237 kubenswrapper[2567]: E0917 17:43:30.104589    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c82f26264ba90782e9fb15f6fbd78bdb798f51c5a2391efa2e98a5f4fe00e43\": container with ID starting with 8c82f26264ba90782e9fb15f6fbd78bdb798f51c5a2391efa2e98a5f4fe00e43 not found: ID does not exist" containerID="8c82f26264ba90782e9fb15f6fbd78bdb798f51c5a2391efa2e98a5f4fe00e43"
Sep 17 17:43:30.104663 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:30.104616    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c82f26264ba90782e9fb15f6fbd78bdb798f51c5a2391efa2e98a5f4fe00e43"} err="failed to get container status \"8c82f26264ba90782e9fb15f6fbd78bdb798f51c5a2391efa2e98a5f4fe00e43\": rpc error: code = NotFound desc = could not find container \"8c82f26264ba90782e9fb15f6fbd78bdb798f51c5a2391efa2e98a5f4fe00e43\": container with ID starting with 8c82f26264ba90782e9fb15f6fbd78bdb798f51c5a2391efa2e98a5f4fe00e43 not found: ID does not exist"
Sep 17 17:43:30.104663 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:30.104630    2567 scope.go:117] "RemoveContainer" containerID="20126bf6b5656fe616b70ec294e7d7a76c3f6012919f3b40668a23c78ea35d40"
Sep 17 17:43:30.104843 ip-10-0-139-237 kubenswrapper[2567]: E0917 17:43:30.104828    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"20126bf6b5656fe616b70ec294e7d7a76c3f6012919f3b40668a23c78ea35d40\": container with ID starting with 20126bf6b5656fe616b70ec294e7d7a76c3f6012919f3b40668a23c78ea35d40 not found: ID does not exist" containerID="20126bf6b5656fe616b70ec294e7d7a76c3f6012919f3b40668a23c78ea35d40"
Sep 17 17:43:30.104885 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:30.104847    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"20126bf6b5656fe616b70ec294e7d7a76c3f6012919f3b40668a23c78ea35d40"} err="failed to get container status \"20126bf6b5656fe616b70ec294e7d7a76c3f6012919f3b40668a23c78ea35d40\": rpc error: code = NotFound desc = could not find container \"20126bf6b5656fe616b70ec294e7d7a76c3f6012919f3b40668a23c78ea35d40\": container with ID starting with 20126bf6b5656fe616b70ec294e7d7a76c3f6012919f3b40668a23c78ea35d40 not found: ID does not exist"
Sep 17 17:43:30.104885 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:30.104858    2567 scope.go:117] "RemoveContainer" containerID="acc68beec13eaaba9fecc2251f46b457e4768bd8cf8d79a1cbae0ed0419a14a1"
Sep 17 17:43:30.105048 ip-10-0-139-237 kubenswrapper[2567]: E0917 17:43:30.105033    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"acc68beec13eaaba9fecc2251f46b457e4768bd8cf8d79a1cbae0ed0419a14a1\": container with ID starting with acc68beec13eaaba9fecc2251f46b457e4768bd8cf8d79a1cbae0ed0419a14a1 not found: ID does not exist" containerID="acc68beec13eaaba9fecc2251f46b457e4768bd8cf8d79a1cbae0ed0419a14a1"
Sep 17 17:43:30.105096 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:30.105051    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"acc68beec13eaaba9fecc2251f46b457e4768bd8cf8d79a1cbae0ed0419a14a1"} err="failed to get container status \"acc68beec13eaaba9fecc2251f46b457e4768bd8cf8d79a1cbae0ed0419a14a1\": rpc error: code = NotFound desc = could not find container \"acc68beec13eaaba9fecc2251f46b457e4768bd8cf8d79a1cbae0ed0419a14a1\": container with ID starting with acc68beec13eaaba9fecc2251f46b457e4768bd8cf8d79a1cbae0ed0419a14a1 not found: ID does not exist"
Sep 17 17:43:30.105096 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:30.105062    2567 scope.go:117] "RemoveContainer" containerID="050a41ae9329c65c8735fa0b973fafba72367f83bd9dd4a8d1ffb6f4575f0830"
Sep 17 17:43:30.105281 ip-10-0-139-237 kubenswrapper[2567]: E0917 17:43:30.105267    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"050a41ae9329c65c8735fa0b973fafba72367f83bd9dd4a8d1ffb6f4575f0830\": container with ID starting with 050a41ae9329c65c8735fa0b973fafba72367f83bd9dd4a8d1ffb6f4575f0830 not found: ID does not exist" containerID="050a41ae9329c65c8735fa0b973fafba72367f83bd9dd4a8d1ffb6f4575f0830"
Sep 17 17:43:30.105324 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:30.105282    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"050a41ae9329c65c8735fa0b973fafba72367f83bd9dd4a8d1ffb6f4575f0830"} err="failed to get container status \"050a41ae9329c65c8735fa0b973fafba72367f83bd9dd4a8d1ffb6f4575f0830\": rpc error: code = NotFound desc = could not find container \"050a41ae9329c65c8735fa0b973fafba72367f83bd9dd4a8d1ffb6f4575f0830\": container with ID starting with 050a41ae9329c65c8735fa0b973fafba72367f83bd9dd4a8d1ffb6f4575f0830 not found: ID does not exist"
Sep 17 17:43:30.105324 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:30.105302    2567 scope.go:117] "RemoveContainer" containerID="273064e4bedf2b1324798b8f2aafaaae9ccb03717d71bb2bda26b9c63085e306"
Sep 17 17:43:30.105470 ip-10-0-139-237 kubenswrapper[2567]: E0917 17:43:30.105449    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"273064e4bedf2b1324798b8f2aafaaae9ccb03717d71bb2bda26b9c63085e306\": container with ID starting with 273064e4bedf2b1324798b8f2aafaaae9ccb03717d71bb2bda26b9c63085e306 not found: ID does not exist" containerID="273064e4bedf2b1324798b8f2aafaaae9ccb03717d71bb2bda26b9c63085e306"
Sep 17 17:43:30.105563 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:30.105473    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"273064e4bedf2b1324798b8f2aafaaae9ccb03717d71bb2bda26b9c63085e306"} err="failed to get container status \"273064e4bedf2b1324798b8f2aafaaae9ccb03717d71bb2bda26b9c63085e306\": rpc error: code = NotFound desc = could not find container \"273064e4bedf2b1324798b8f2aafaaae9ccb03717d71bb2bda26b9c63085e306\": container with ID starting with 273064e4bedf2b1324798b8f2aafaaae9ccb03717d71bb2bda26b9c63085e306 not found: ID does not exist"
Sep 17 17:43:30.105563 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:30.105484    2567 scope.go:117] "RemoveContainer" containerID="3d98a78d4042469903bb7f4870086e12672504526d45866281542e69841caf04"
Sep 17 17:43:30.105714 ip-10-0-139-237 kubenswrapper[2567]: E0917 17:43:30.105698    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d98a78d4042469903bb7f4870086e12672504526d45866281542e69841caf04\": container with ID starting with 3d98a78d4042469903bb7f4870086e12672504526d45866281542e69841caf04 not found: ID does not exist" containerID="3d98a78d4042469903bb7f4870086e12672504526d45866281542e69841caf04"
Sep 17 17:43:30.105755 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:30.105717    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d98a78d4042469903bb7f4870086e12672504526d45866281542e69841caf04"} err="failed to get container status \"3d98a78d4042469903bb7f4870086e12672504526d45866281542e69841caf04\": rpc error: code = NotFound desc = could not find container \"3d98a78d4042469903bb7f4870086e12672504526d45866281542e69841caf04\": container with ID starting with 3d98a78d4042469903bb7f4870086e12672504526d45866281542e69841caf04 not found: ID does not exist"
Sep 17 17:43:30.105755 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:30.105729    2567 scope.go:117] "RemoveContainer" containerID="d9543b6bf84dd3ea06ca00cc9480684d94b52827b927f3e39c5279872c7fbae1"
Sep 17 17:43:30.105943 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:30.105925    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d9543b6bf84dd3ea06ca00cc9480684d94b52827b927f3e39c5279872c7fbae1"} err="failed to get container status \"d9543b6bf84dd3ea06ca00cc9480684d94b52827b927f3e39c5279872c7fbae1\": rpc error: code = NotFound desc = could not find container \"d9543b6bf84dd3ea06ca00cc9480684d94b52827b927f3e39c5279872c7fbae1\": container with ID starting with d9543b6bf84dd3ea06ca00cc9480684d94b52827b927f3e39c5279872c7fbae1 not found: ID does not exist"
Sep 17 17:43:30.105988 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:30.105945    2567 scope.go:117] "RemoveContainer" containerID="8c82f26264ba90782e9fb15f6fbd78bdb798f51c5a2391efa2e98a5f4fe00e43"
Sep 17 17:43:30.106202 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:30.106182    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c82f26264ba90782e9fb15f6fbd78bdb798f51c5a2391efa2e98a5f4fe00e43"} err="failed to get container status \"8c82f26264ba90782e9fb15f6fbd78bdb798f51c5a2391efa2e98a5f4fe00e43\": rpc error: code = NotFound desc = could not find container \"8c82f26264ba90782e9fb15f6fbd78bdb798f51c5a2391efa2e98a5f4fe00e43\": container with ID starting with 8c82f26264ba90782e9fb15f6fbd78bdb798f51c5a2391efa2e98a5f4fe00e43 not found: ID does not exist"
Sep 17 17:43:30.106276 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:30.106213    2567 scope.go:117] "RemoveContainer" containerID="20126bf6b5656fe616b70ec294e7d7a76c3f6012919f3b40668a23c78ea35d40"
Sep 17 17:43:30.106438 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:30.106424    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"20126bf6b5656fe616b70ec294e7d7a76c3f6012919f3b40668a23c78ea35d40"} err="failed to get container status \"20126bf6b5656fe616b70ec294e7d7a76c3f6012919f3b40668a23c78ea35d40\": rpc error: code = NotFound desc = could not find container \"20126bf6b5656fe616b70ec294e7d7a76c3f6012919f3b40668a23c78ea35d40\": container with ID starting with 20126bf6b5656fe616b70ec294e7d7a76c3f6012919f3b40668a23c78ea35d40 not found: ID does not exist"
Sep 17 17:43:30.106481 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:30.106439    2567 scope.go:117] "RemoveContainer" containerID="acc68beec13eaaba9fecc2251f46b457e4768bd8cf8d79a1cbae0ed0419a14a1"
Sep 17 17:43:30.106680 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:30.106659    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"acc68beec13eaaba9fecc2251f46b457e4768bd8cf8d79a1cbae0ed0419a14a1"} err="failed to get container status \"acc68beec13eaaba9fecc2251f46b457e4768bd8cf8d79a1cbae0ed0419a14a1\": rpc error: code = NotFound desc = could not find container \"acc68beec13eaaba9fecc2251f46b457e4768bd8cf8d79a1cbae0ed0419a14a1\": container with ID starting with acc68beec13eaaba9fecc2251f46b457e4768bd8cf8d79a1cbae0ed0419a14a1 not found: ID does not exist"
Sep 17 17:43:30.106680 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:30.106679    2567 scope.go:117] "RemoveContainer" containerID="050a41ae9329c65c8735fa0b973fafba72367f83bd9dd4a8d1ffb6f4575f0830"
Sep 17 17:43:30.106885 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:30.106867    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"050a41ae9329c65c8735fa0b973fafba72367f83bd9dd4a8d1ffb6f4575f0830"} err="failed to get container status \"050a41ae9329c65c8735fa0b973fafba72367f83bd9dd4a8d1ffb6f4575f0830\": rpc error: code = NotFound desc = could not find container \"050a41ae9329c65c8735fa0b973fafba72367f83bd9dd4a8d1ffb6f4575f0830\": container with ID starting with 050a41ae9329c65c8735fa0b973fafba72367f83bd9dd4a8d1ffb6f4575f0830 not found: ID does not exist"
Sep 17 17:43:30.106954 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:30.106886    2567 scope.go:117] "RemoveContainer" containerID="273064e4bedf2b1324798b8f2aafaaae9ccb03717d71bb2bda26b9c63085e306"
Sep 17 17:43:30.107149 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:30.107131    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"273064e4bedf2b1324798b8f2aafaaae9ccb03717d71bb2bda26b9c63085e306"} err="failed to get container status \"273064e4bedf2b1324798b8f2aafaaae9ccb03717d71bb2bda26b9c63085e306\": rpc error: code = NotFound desc = could not find container \"273064e4bedf2b1324798b8f2aafaaae9ccb03717d71bb2bda26b9c63085e306\": container with ID starting with 273064e4bedf2b1324798b8f2aafaaae9ccb03717d71bb2bda26b9c63085e306 not found: ID does not exist"
Sep 17 17:43:30.107215 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:30.107150    2567 scope.go:117] "RemoveContainer" containerID="3d98a78d4042469903bb7f4870086e12672504526d45866281542e69841caf04"
Sep 17 17:43:30.107403 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:30.107384    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d98a78d4042469903bb7f4870086e12672504526d45866281542e69841caf04"} err="failed to get container status \"3d98a78d4042469903bb7f4870086e12672504526d45866281542e69841caf04\": rpc error: code = NotFound desc = could not find container \"3d98a78d4042469903bb7f4870086e12672504526d45866281542e69841caf04\": container with ID starting with 3d98a78d4042469903bb7f4870086e12672504526d45866281542e69841caf04 not found: ID does not exist"
Sep 17 17:43:30.107403 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:30.107402    2567 scope.go:117] "RemoveContainer" containerID="d9543b6bf84dd3ea06ca00cc9480684d94b52827b927f3e39c5279872c7fbae1"
Sep 17 17:43:30.107737 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:30.107706    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d9543b6bf84dd3ea06ca00cc9480684d94b52827b927f3e39c5279872c7fbae1"} err="failed to get container status \"d9543b6bf84dd3ea06ca00cc9480684d94b52827b927f3e39c5279872c7fbae1\": rpc error: code = NotFound desc = could not find container \"d9543b6bf84dd3ea06ca00cc9480684d94b52827b927f3e39c5279872c7fbae1\": container with ID starting with d9543b6bf84dd3ea06ca00cc9480684d94b52827b927f3e39c5279872c7fbae1 not found: ID does not exist"
Sep 17 17:43:30.107824 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:30.107739    2567 scope.go:117] "RemoveContainer" containerID="8c82f26264ba90782e9fb15f6fbd78bdb798f51c5a2391efa2e98a5f4fe00e43"
Sep 17 17:43:30.108121 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:30.108098    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c82f26264ba90782e9fb15f6fbd78bdb798f51c5a2391efa2e98a5f4fe00e43"} err="failed to get container status \"8c82f26264ba90782e9fb15f6fbd78bdb798f51c5a2391efa2e98a5f4fe00e43\": rpc error: code = NotFound desc = could not find container \"8c82f26264ba90782e9fb15f6fbd78bdb798f51c5a2391efa2e98a5f4fe00e43\": container with ID starting with 8c82f26264ba90782e9fb15f6fbd78bdb798f51c5a2391efa2e98a5f4fe00e43 not found: ID does not exist"
Sep 17 17:43:30.108202 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:30.108122    2567 scope.go:117] "RemoveContainer" containerID="20126bf6b5656fe616b70ec294e7d7a76c3f6012919f3b40668a23c78ea35d40"
Sep 17 17:43:30.108359 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:30.108344    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"20126bf6b5656fe616b70ec294e7d7a76c3f6012919f3b40668a23c78ea35d40"} err="failed to get container status \"20126bf6b5656fe616b70ec294e7d7a76c3f6012919f3b40668a23c78ea35d40\": rpc error: code = NotFound desc = could not find container \"20126bf6b5656fe616b70ec294e7d7a76c3f6012919f3b40668a23c78ea35d40\": container with ID starting with 20126bf6b5656fe616b70ec294e7d7a76c3f6012919f3b40668a23c78ea35d40 not found: ID does not exist"
Sep 17 17:43:30.108401 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:30.108359    2567 scope.go:117] "RemoveContainer" containerID="acc68beec13eaaba9fecc2251f46b457e4768bd8cf8d79a1cbae0ed0419a14a1"
Sep 17 17:43:30.108595 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:30.108579    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"acc68beec13eaaba9fecc2251f46b457e4768bd8cf8d79a1cbae0ed0419a14a1"} err="failed to get container status \"acc68beec13eaaba9fecc2251f46b457e4768bd8cf8d79a1cbae0ed0419a14a1\": rpc error: code = NotFound desc = could not find container \"acc68beec13eaaba9fecc2251f46b457e4768bd8cf8d79a1cbae0ed0419a14a1\": container with ID starting with acc68beec13eaaba9fecc2251f46b457e4768bd8cf8d79a1cbae0ed0419a14a1 not found: ID does not exist"
Sep 17 17:43:30.108645 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:30.108594    2567 scope.go:117] "RemoveContainer" containerID="050a41ae9329c65c8735fa0b973fafba72367f83bd9dd4a8d1ffb6f4575f0830"
Sep 17 17:43:30.108801 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:30.108786    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"050a41ae9329c65c8735fa0b973fafba72367f83bd9dd4a8d1ffb6f4575f0830"} err="failed to get container status \"050a41ae9329c65c8735fa0b973fafba72367f83bd9dd4a8d1ffb6f4575f0830\": rpc error: code = NotFound desc = could not find container \"050a41ae9329c65c8735fa0b973fafba72367f83bd9dd4a8d1ffb6f4575f0830\": container with ID starting with 050a41ae9329c65c8735fa0b973fafba72367f83bd9dd4a8d1ffb6f4575f0830 not found: ID does not exist"
Sep 17 17:43:30.108843 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:30.108802    2567 scope.go:117] "RemoveContainer" containerID="273064e4bedf2b1324798b8f2aafaaae9ccb03717d71bb2bda26b9c63085e306"
Sep 17 17:43:30.109009 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:30.108994    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"273064e4bedf2b1324798b8f2aafaaae9ccb03717d71bb2bda26b9c63085e306"} err="failed to get container status \"273064e4bedf2b1324798b8f2aafaaae9ccb03717d71bb2bda26b9c63085e306\": rpc error: code = NotFound desc = could not find container \"273064e4bedf2b1324798b8f2aafaaae9ccb03717d71bb2bda26b9c63085e306\": container with ID starting with 273064e4bedf2b1324798b8f2aafaaae9ccb03717d71bb2bda26b9c63085e306 not found: ID does not exist"
Sep 17 17:43:30.109057 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:30.109009    2567 scope.go:117] "RemoveContainer" containerID="3d98a78d4042469903bb7f4870086e12672504526d45866281542e69841caf04"
Sep 17 17:43:30.109198 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:30.109184    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d98a78d4042469903bb7f4870086e12672504526d45866281542e69841caf04"} err="failed to get container status \"3d98a78d4042469903bb7f4870086e12672504526d45866281542e69841caf04\": rpc error: code = NotFound desc = could not find container \"3d98a78d4042469903bb7f4870086e12672504526d45866281542e69841caf04\": container with ID starting with 3d98a78d4042469903bb7f4870086e12672504526d45866281542e69841caf04 not found: ID does not exist"
Sep 17 17:43:30.109244 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:30.109197    2567 scope.go:117] "RemoveContainer" containerID="d9543b6bf84dd3ea06ca00cc9480684d94b52827b927f3e39c5279872c7fbae1"
Sep 17 17:43:30.109376 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:30.109362    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d9543b6bf84dd3ea06ca00cc9480684d94b52827b927f3e39c5279872c7fbae1"} err="failed to get container status \"d9543b6bf84dd3ea06ca00cc9480684d94b52827b927f3e39c5279872c7fbae1\": rpc error: code = NotFound desc = could not find container \"d9543b6bf84dd3ea06ca00cc9480684d94b52827b927f3e39c5279872c7fbae1\": container with ID starting with d9543b6bf84dd3ea06ca00cc9480684d94b52827b927f3e39c5279872c7fbae1 not found: ID does not exist"
Sep 17 17:43:30.109420 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:30.109375    2567 scope.go:117] "RemoveContainer" containerID="8c82f26264ba90782e9fb15f6fbd78bdb798f51c5a2391efa2e98a5f4fe00e43"
Sep 17 17:43:30.109663 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:30.109636    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c82f26264ba90782e9fb15f6fbd78bdb798f51c5a2391efa2e98a5f4fe00e43"} err="failed to get container status \"8c82f26264ba90782e9fb15f6fbd78bdb798f51c5a2391efa2e98a5f4fe00e43\": rpc error: code = NotFound desc = could not find container \"8c82f26264ba90782e9fb15f6fbd78bdb798f51c5a2391efa2e98a5f4fe00e43\": container with ID starting with 8c82f26264ba90782e9fb15f6fbd78bdb798f51c5a2391efa2e98a5f4fe00e43 not found: ID does not exist"
Sep 17 17:43:30.109712 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:30.109663    2567 scope.go:117] "RemoveContainer" containerID="20126bf6b5656fe616b70ec294e7d7a76c3f6012919f3b40668a23c78ea35d40"
Sep 17 17:43:30.109901 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:30.109882    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"20126bf6b5656fe616b70ec294e7d7a76c3f6012919f3b40668a23c78ea35d40"} err="failed to get container status \"20126bf6b5656fe616b70ec294e7d7a76c3f6012919f3b40668a23c78ea35d40\": rpc error: code = NotFound desc = could not find container \"20126bf6b5656fe616b70ec294e7d7a76c3f6012919f3b40668a23c78ea35d40\": container with ID starting with 20126bf6b5656fe616b70ec294e7d7a76c3f6012919f3b40668a23c78ea35d40 not found: ID does not exist"
Sep 17 17:43:30.109946 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:30.109900    2567 scope.go:117] "RemoveContainer" containerID="acc68beec13eaaba9fecc2251f46b457e4768bd8cf8d79a1cbae0ed0419a14a1"
Sep 17 17:43:30.110080 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:30.110063    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"acc68beec13eaaba9fecc2251f46b457e4768bd8cf8d79a1cbae0ed0419a14a1"} err="failed to get container status \"acc68beec13eaaba9fecc2251f46b457e4768bd8cf8d79a1cbae0ed0419a14a1\": rpc error: code = NotFound desc = could not find container \"acc68beec13eaaba9fecc2251f46b457e4768bd8cf8d79a1cbae0ed0419a14a1\": container with ID starting with acc68beec13eaaba9fecc2251f46b457e4768bd8cf8d79a1cbae0ed0419a14a1 not found: ID does not exist"
Sep 17 17:43:30.110133 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:30.110079    2567 scope.go:117] "RemoveContainer" containerID="050a41ae9329c65c8735fa0b973fafba72367f83bd9dd4a8d1ffb6f4575f0830"
Sep 17 17:43:30.110294 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:30.110266    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"050a41ae9329c65c8735fa0b973fafba72367f83bd9dd4a8d1ffb6f4575f0830"} err="failed to get container status \"050a41ae9329c65c8735fa0b973fafba72367f83bd9dd4a8d1ffb6f4575f0830\": rpc error: code = NotFound desc = could not find container \"050a41ae9329c65c8735fa0b973fafba72367f83bd9dd4a8d1ffb6f4575f0830\": container with ID starting with 050a41ae9329c65c8735fa0b973fafba72367f83bd9dd4a8d1ffb6f4575f0830 not found: ID does not exist"
Sep 17 17:43:30.110294 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:30.110292    2567 scope.go:117] "RemoveContainer" containerID="273064e4bedf2b1324798b8f2aafaaae9ccb03717d71bb2bda26b9c63085e306"
Sep 17 17:43:30.110550 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:30.110521    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"273064e4bedf2b1324798b8f2aafaaae9ccb03717d71bb2bda26b9c63085e306"} err="failed to get container status \"273064e4bedf2b1324798b8f2aafaaae9ccb03717d71bb2bda26b9c63085e306\": rpc error: code = NotFound desc = could not find container \"273064e4bedf2b1324798b8f2aafaaae9ccb03717d71bb2bda26b9c63085e306\": container with ID starting with 273064e4bedf2b1324798b8f2aafaaae9ccb03717d71bb2bda26b9c63085e306 not found: ID does not exist"
Sep 17 17:43:30.110592 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:30.110553    2567 scope.go:117] "RemoveContainer" containerID="3d98a78d4042469903bb7f4870086e12672504526d45866281542e69841caf04"
Sep 17 17:43:30.110773 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:30.110756    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d98a78d4042469903bb7f4870086e12672504526d45866281542e69841caf04"} err="failed to get container status \"3d98a78d4042469903bb7f4870086e12672504526d45866281542e69841caf04\": rpc error: code = NotFound desc = could not find container \"3d98a78d4042469903bb7f4870086e12672504526d45866281542e69841caf04\": container with ID starting with 3d98a78d4042469903bb7f4870086e12672504526d45866281542e69841caf04 not found: ID does not exist"
Sep 17 17:43:30.110816 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:30.110773    2567 scope.go:117] "RemoveContainer" containerID="d9543b6bf84dd3ea06ca00cc9480684d94b52827b927f3e39c5279872c7fbae1"
Sep 17 17:43:30.110995 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:30.110981    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d9543b6bf84dd3ea06ca00cc9480684d94b52827b927f3e39c5279872c7fbae1"} err="failed to get container status \"d9543b6bf84dd3ea06ca00cc9480684d94b52827b927f3e39c5279872c7fbae1\": rpc error: code = NotFound desc = could not find container \"d9543b6bf84dd3ea06ca00cc9480684d94b52827b927f3e39c5279872c7fbae1\": container with ID starting with d9543b6bf84dd3ea06ca00cc9480684d94b52827b927f3e39c5279872c7fbae1 not found: ID does not exist"
Sep 17 17:43:30.111036 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:30.110995    2567 scope.go:117] "RemoveContainer" containerID="8c82f26264ba90782e9fb15f6fbd78bdb798f51c5a2391efa2e98a5f4fe00e43"
Sep 17 17:43:30.111227 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:30.111207    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c82f26264ba90782e9fb15f6fbd78bdb798f51c5a2391efa2e98a5f4fe00e43"} err="failed to get container status \"8c82f26264ba90782e9fb15f6fbd78bdb798f51c5a2391efa2e98a5f4fe00e43\": rpc error: code = NotFound desc = could not find container \"8c82f26264ba90782e9fb15f6fbd78bdb798f51c5a2391efa2e98a5f4fe00e43\": container with ID starting with 8c82f26264ba90782e9fb15f6fbd78bdb798f51c5a2391efa2e98a5f4fe00e43 not found: ID does not exist"
Sep 17 17:43:30.111283 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:30.111228    2567 scope.go:117] "RemoveContainer" containerID="20126bf6b5656fe616b70ec294e7d7a76c3f6012919f3b40668a23c78ea35d40"
Sep 17 17:43:30.111458 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:30.111439    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"20126bf6b5656fe616b70ec294e7d7a76c3f6012919f3b40668a23c78ea35d40"} err="failed to get container status \"20126bf6b5656fe616b70ec294e7d7a76c3f6012919f3b40668a23c78ea35d40\": rpc error: code = NotFound desc = could not find container \"20126bf6b5656fe616b70ec294e7d7a76c3f6012919f3b40668a23c78ea35d40\": container with ID starting with 20126bf6b5656fe616b70ec294e7d7a76c3f6012919f3b40668a23c78ea35d40 not found: ID does not exist"
Sep 17 17:43:30.111496 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:30.111459    2567 scope.go:117] "RemoveContainer" containerID="acc68beec13eaaba9fecc2251f46b457e4768bd8cf8d79a1cbae0ed0419a14a1"
Sep 17 17:43:30.111700 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:30.111685    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"acc68beec13eaaba9fecc2251f46b457e4768bd8cf8d79a1cbae0ed0419a14a1"} err="failed to get container status \"acc68beec13eaaba9fecc2251f46b457e4768bd8cf8d79a1cbae0ed0419a14a1\": rpc error: code = NotFound desc = could not find container \"acc68beec13eaaba9fecc2251f46b457e4768bd8cf8d79a1cbae0ed0419a14a1\": container with ID starting with acc68beec13eaaba9fecc2251f46b457e4768bd8cf8d79a1cbae0ed0419a14a1 not found: ID does not exist"
Sep 17 17:43:30.111738 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:30.111702    2567 scope.go:117] "RemoveContainer" containerID="050a41ae9329c65c8735fa0b973fafba72367f83bd9dd4a8d1ffb6f4575f0830"
Sep 17 17:43:30.111937 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:30.111920    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"050a41ae9329c65c8735fa0b973fafba72367f83bd9dd4a8d1ffb6f4575f0830"} err="failed to get container status \"050a41ae9329c65c8735fa0b973fafba72367f83bd9dd4a8d1ffb6f4575f0830\": rpc error: code = NotFound desc = could not find container \"050a41ae9329c65c8735fa0b973fafba72367f83bd9dd4a8d1ffb6f4575f0830\": container with ID starting with 050a41ae9329c65c8735fa0b973fafba72367f83bd9dd4a8d1ffb6f4575f0830 not found: ID does not exist"
Sep 17 17:43:30.111983 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:30.111937    2567 scope.go:117] "RemoveContainer" containerID="273064e4bedf2b1324798b8f2aafaaae9ccb03717d71bb2bda26b9c63085e306"
Sep 17 17:43:30.112184 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:30.112158    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"273064e4bedf2b1324798b8f2aafaaae9ccb03717d71bb2bda26b9c63085e306"} err="failed to get container status \"273064e4bedf2b1324798b8f2aafaaae9ccb03717d71bb2bda26b9c63085e306\": rpc error: code = NotFound desc = could not find container \"273064e4bedf2b1324798b8f2aafaaae9ccb03717d71bb2bda26b9c63085e306\": container with ID starting with 273064e4bedf2b1324798b8f2aafaaae9ccb03717d71bb2bda26b9c63085e306 not found: ID does not exist"
Sep 17 17:43:30.112232 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:30.112184    2567 scope.go:117] "RemoveContainer" containerID="3d98a78d4042469903bb7f4870086e12672504526d45866281542e69841caf04"
Sep 17 17:43:30.112424 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:30.112404    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d98a78d4042469903bb7f4870086e12672504526d45866281542e69841caf04"} err="failed to get container status \"3d98a78d4042469903bb7f4870086e12672504526d45866281542e69841caf04\": rpc error: code = NotFound desc = could not find container \"3d98a78d4042469903bb7f4870086e12672504526d45866281542e69841caf04\": container with ID starting with 3d98a78d4042469903bb7f4870086e12672504526d45866281542e69841caf04 not found: ID does not exist"
Sep 17 17:43:30.112491 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:30.112426    2567 scope.go:117] "RemoveContainer" containerID="d9543b6bf84dd3ea06ca00cc9480684d94b52827b927f3e39c5279872c7fbae1"
Sep 17 17:43:30.112695 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:30.112679    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d9543b6bf84dd3ea06ca00cc9480684d94b52827b927f3e39c5279872c7fbae1"} err="failed to get container status \"d9543b6bf84dd3ea06ca00cc9480684d94b52827b927f3e39c5279872c7fbae1\": rpc error: code = NotFound desc = could not find container \"d9543b6bf84dd3ea06ca00cc9480684d94b52827b927f3e39c5279872c7fbae1\": container with ID starting with d9543b6bf84dd3ea06ca00cc9480684d94b52827b927f3e39c5279872c7fbae1 not found: ID does not exist"
Sep 17 17:43:30.112743 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:30.112703    2567 scope.go:117] "RemoveContainer" containerID="8c82f26264ba90782e9fb15f6fbd78bdb798f51c5a2391efa2e98a5f4fe00e43"
Sep 17 17:43:30.112939 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:30.112920    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c82f26264ba90782e9fb15f6fbd78bdb798f51c5a2391efa2e98a5f4fe00e43"} err="failed to get container status \"8c82f26264ba90782e9fb15f6fbd78bdb798f51c5a2391efa2e98a5f4fe00e43\": rpc error: code = NotFound desc = could not find container \"8c82f26264ba90782e9fb15f6fbd78bdb798f51c5a2391efa2e98a5f4fe00e43\": container with ID starting with 8c82f26264ba90782e9fb15f6fbd78bdb798f51c5a2391efa2e98a5f4fe00e43 not found: ID does not exist"
Sep 17 17:43:30.113006 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:30.112941    2567 scope.go:117] "RemoveContainer" containerID="20126bf6b5656fe616b70ec294e7d7a76c3f6012919f3b40668a23c78ea35d40"
Sep 17 17:43:30.113155 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:30.113135    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"20126bf6b5656fe616b70ec294e7d7a76c3f6012919f3b40668a23c78ea35d40"} err="failed to get container status \"20126bf6b5656fe616b70ec294e7d7a76c3f6012919f3b40668a23c78ea35d40\": rpc error: code = NotFound desc = could not find container \"20126bf6b5656fe616b70ec294e7d7a76c3f6012919f3b40668a23c78ea35d40\": container with ID starting with 20126bf6b5656fe616b70ec294e7d7a76c3f6012919f3b40668a23c78ea35d40 not found: ID does not exist"
Sep 17 17:43:30.113198 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:30.113156    2567 scope.go:117] "RemoveContainer" containerID="acc68beec13eaaba9fecc2251f46b457e4768bd8cf8d79a1cbae0ed0419a14a1"
Sep 17 17:43:30.113343 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:30.113329    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"acc68beec13eaaba9fecc2251f46b457e4768bd8cf8d79a1cbae0ed0419a14a1"} err="failed to get container status \"acc68beec13eaaba9fecc2251f46b457e4768bd8cf8d79a1cbae0ed0419a14a1\": rpc error: code = NotFound desc = could not find container \"acc68beec13eaaba9fecc2251f46b457e4768bd8cf8d79a1cbae0ed0419a14a1\": container with ID starting with acc68beec13eaaba9fecc2251f46b457e4768bd8cf8d79a1cbae0ed0419a14a1 not found: ID does not exist"
Sep 17 17:43:30.113343 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:30.113343    2567 scope.go:117] "RemoveContainer" containerID="050a41ae9329c65c8735fa0b973fafba72367f83bd9dd4a8d1ffb6f4575f0830"
Sep 17 17:43:30.113582 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:30.113566    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"050a41ae9329c65c8735fa0b973fafba72367f83bd9dd4a8d1ffb6f4575f0830"} err="failed to get container status \"050a41ae9329c65c8735fa0b973fafba72367f83bd9dd4a8d1ffb6f4575f0830\": rpc error: code = NotFound desc = could not find container \"050a41ae9329c65c8735fa0b973fafba72367f83bd9dd4a8d1ffb6f4575f0830\": container with ID starting with 050a41ae9329c65c8735fa0b973fafba72367f83bd9dd4a8d1ffb6f4575f0830 not found: ID does not exist"
Sep 17 17:43:30.113634 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:30.113582    2567 scope.go:117] "RemoveContainer" containerID="273064e4bedf2b1324798b8f2aafaaae9ccb03717d71bb2bda26b9c63085e306"
Sep 17 17:43:30.113823 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:30.113796    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"273064e4bedf2b1324798b8f2aafaaae9ccb03717d71bb2bda26b9c63085e306"} err="failed to get container status \"273064e4bedf2b1324798b8f2aafaaae9ccb03717d71bb2bda26b9c63085e306\": rpc error: code = NotFound desc = could not find container \"273064e4bedf2b1324798b8f2aafaaae9ccb03717d71bb2bda26b9c63085e306\": container with ID starting with 273064e4bedf2b1324798b8f2aafaaae9ccb03717d71bb2bda26b9c63085e306 not found: ID does not exist"
Sep 17 17:43:30.113870 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:30.113824    2567 scope.go:117] "RemoveContainer" containerID="3d98a78d4042469903bb7f4870086e12672504526d45866281542e69841caf04"
Sep 17 17:43:30.114041 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:30.114025    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d98a78d4042469903bb7f4870086e12672504526d45866281542e69841caf04"} err="failed to get container status \"3d98a78d4042469903bb7f4870086e12672504526d45866281542e69841caf04\": rpc error: code = NotFound desc = could not find container \"3d98a78d4042469903bb7f4870086e12672504526d45866281542e69841caf04\": container with ID starting with 3d98a78d4042469903bb7f4870086e12672504526d45866281542e69841caf04 not found: ID does not exist"
Sep 17 17:43:30.118831 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:30.118812    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 17 17:43:30.119093 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:30.119081    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8dcb3c07-9ad2-4670-b801-0acce83c6faf" containerName="init-config-reloader"
Sep 17 17:43:30.119130 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:30.119095    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="8dcb3c07-9ad2-4670-b801-0acce83c6faf" containerName="init-config-reloader"
Sep 17 17:43:30.119130 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:30.119105    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8dcb3c07-9ad2-4670-b801-0acce83c6faf" containerName="thanos-sidecar"
Sep 17 17:43:30.119130 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:30.119111    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="8dcb3c07-9ad2-4670-b801-0acce83c6faf" containerName="thanos-sidecar"
Sep 17 17:43:30.119130 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:30.119122    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8dcb3c07-9ad2-4670-b801-0acce83c6faf" containerName="prometheus"
Sep 17 17:43:30.119130 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:30.119128    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="8dcb3c07-9ad2-4670-b801-0acce83c6faf" containerName="prometheus"
Sep 17 17:43:30.119325 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:30.119137    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8dcb3c07-9ad2-4670-b801-0acce83c6faf" containerName="config-reloader"
Sep 17 17:43:30.119325 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:30.119142    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="8dcb3c07-9ad2-4670-b801-0acce83c6faf" containerName="config-reloader"
Sep 17 17:43:30.119325 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:30.119156    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8dcb3c07-9ad2-4670-b801-0acce83c6faf" containerName="kube-rbac-proxy"
Sep 17 17:43:30.119325 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:30.119163    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="8dcb3c07-9ad2-4670-b801-0acce83c6faf" containerName="kube-rbac-proxy"
Sep 17 17:43:30.119325 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:30.119174    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8dcb3c07-9ad2-4670-b801-0acce83c6faf" containerName="kube-rbac-proxy-thanos"
Sep 17 17:43:30.119325 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:30.119179    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="8dcb3c07-9ad2-4670-b801-0acce83c6faf" containerName="kube-rbac-proxy-thanos"
Sep 17 17:43:30.119325 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:30.119185    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8dcb3c07-9ad2-4670-b801-0acce83c6faf" containerName="kube-rbac-proxy-web"
Sep 17 17:43:30.119325 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:30.119190    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="8dcb3c07-9ad2-4670-b801-0acce83c6faf" containerName="kube-rbac-proxy-web"
Sep 17 17:43:30.119325 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:30.119232    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="8dcb3c07-9ad2-4670-b801-0acce83c6faf" containerName="kube-rbac-proxy-web"
Sep 17 17:43:30.119325 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:30.119241    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="8dcb3c07-9ad2-4670-b801-0acce83c6faf" containerName="prometheus"
Sep 17 17:43:30.119325 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:30.119249    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="8dcb3c07-9ad2-4670-b801-0acce83c6faf" containerName="kube-rbac-proxy"
Sep 17 17:43:30.119325 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:30.119254    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="8dcb3c07-9ad2-4670-b801-0acce83c6faf" containerName="thanos-sidecar"
Sep 17 17:43:30.119325 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:30.119259    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="8dcb3c07-9ad2-4670-b801-0acce83c6faf" containerName="kube-rbac-proxy-thanos"
Sep 17 17:43:30.119325 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:30.119264    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="8dcb3c07-9ad2-4670-b801-0acce83c6faf" containerName="config-reloader"
Sep 17 17:43:30.122940 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:30.122922    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:43:30.126493 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:30.126471    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 17 17:43:30.127121 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:30.127103    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 17 17:43:30.127323 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:30.127306    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 17 17:43:30.127430 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:30.127414    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 17 17:43:30.127503 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:30.127490    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-dgttk\""
Sep 17 17:43:30.127620 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:30.127602    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 17 17:43:30.127676 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:30.127660    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 17 17:43:30.127855 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:30.127841    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-flrbqlhtmqr18\""
Sep 17 17:43:30.128286 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:30.128268    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 17 17:43:30.129713 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:30.129658    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 17 17:43:30.130809 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:30.130792    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 17 17:43:30.130910 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:30.130894    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Sep 17 17:43:30.130985 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:30.130970    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 17 17:43:30.130985 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:30.130975    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 17 17:43:30.131752 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:30.131733    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 17 17:43:30.134422 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:30.134405    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 17 17:43:30.136660 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:30.136644    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 17 17:43:30.140449 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:30.140429    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 17 17:43:30.224350 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:30.224315    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bfeb358d-808a-4246-a3d4-35add25eef8a-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"bfeb358d-808a-4246-a3d4-35add25eef8a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:43:30.224350 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:30.224352    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/bfeb358d-808a-4246-a3d4-35add25eef8a-config\") pod \"prometheus-k8s-0\" (UID: \"bfeb358d-808a-4246-a3d4-35add25eef8a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:43:30.224566 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:30.224373    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bfeb358d-808a-4246-a3d4-35add25eef8a-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"bfeb358d-808a-4246-a3d4-35add25eef8a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:43:30.224566 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:30.224390    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/bfeb358d-808a-4246-a3d4-35add25eef8a-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"bfeb358d-808a-4246-a3d4-35add25eef8a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:43:30.224566 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:30.224463    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/bfeb358d-808a-4246-a3d4-35add25eef8a-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"bfeb358d-808a-4246-a3d4-35add25eef8a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:43:30.224566 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:30.224493    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/bfeb358d-808a-4246-a3d4-35add25eef8a-config-out\") pod \"prometheus-k8s-0\" (UID: \"bfeb358d-808a-4246-a3d4-35add25eef8a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:43:30.224566 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:30.224511    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/bfeb358d-808a-4246-a3d4-35add25eef8a-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"bfeb358d-808a-4246-a3d4-35add25eef8a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:43:30.224566 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:30.224552    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/bfeb358d-808a-4246-a3d4-35add25eef8a-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"bfeb358d-808a-4246-a3d4-35add25eef8a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:43:30.224754 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:30.224572    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/bfeb358d-808a-4246-a3d4-35add25eef8a-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"bfeb358d-808a-4246-a3d4-35add25eef8a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:43:30.224754 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:30.224591    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bfeb358d-808a-4246-a3d4-35add25eef8a-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"bfeb358d-808a-4246-a3d4-35add25eef8a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:43:30.224754 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:30.224608    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/bfeb358d-808a-4246-a3d4-35add25eef8a-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"bfeb358d-808a-4246-a3d4-35add25eef8a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:43:30.224754 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:30.224627    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/bfeb358d-808a-4246-a3d4-35add25eef8a-web-config\") pod \"prometheus-k8s-0\" (UID: \"bfeb358d-808a-4246-a3d4-35add25eef8a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:43:30.224754 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:30.224673    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/bfeb358d-808a-4246-a3d4-35add25eef8a-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"bfeb358d-808a-4246-a3d4-35add25eef8a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:43:30.224754 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:30.224721    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/bfeb358d-808a-4246-a3d4-35add25eef8a-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"bfeb358d-808a-4246-a3d4-35add25eef8a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:43:30.224754 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:30.224747    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lgh9m\" (UniqueName: \"kubernetes.io/projected/bfeb358d-808a-4246-a3d4-35add25eef8a-kube-api-access-lgh9m\") pod \"prometheus-k8s-0\" (UID: \"bfeb358d-808a-4246-a3d4-35add25eef8a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:43:30.224991 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:30.224793    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/bfeb358d-808a-4246-a3d4-35add25eef8a-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"bfeb358d-808a-4246-a3d4-35add25eef8a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:43:30.224991 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:30.224818    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bfeb358d-808a-4246-a3d4-35add25eef8a-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"bfeb358d-808a-4246-a3d4-35add25eef8a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:43:30.224991 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:30.224841    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/bfeb358d-808a-4246-a3d4-35add25eef8a-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"bfeb358d-808a-4246-a3d4-35add25eef8a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:43:30.224991 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:30.224874    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/bfeb358d-808a-4246-a3d4-35add25eef8a-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"bfeb358d-808a-4246-a3d4-35add25eef8a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:43:30.224991 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:30.224908    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/bfeb358d-808a-4246-a3d4-35add25eef8a-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"bfeb358d-808a-4246-a3d4-35add25eef8a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:43:30.325785 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:30.325751    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/bfeb358d-808a-4246-a3d4-35add25eef8a-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"bfeb358d-808a-4246-a3d4-35add25eef8a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:43:30.325963 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:30.325791    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bfeb358d-808a-4246-a3d4-35add25eef8a-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"bfeb358d-808a-4246-a3d4-35add25eef8a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:43:30.325963 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:30.325827    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/bfeb358d-808a-4246-a3d4-35add25eef8a-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"bfeb358d-808a-4246-a3d4-35add25eef8a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:43:30.325963 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:30.325848    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/bfeb358d-808a-4246-a3d4-35add25eef8a-web-config\") pod \"prometheus-k8s-0\" (UID: \"bfeb358d-808a-4246-a3d4-35add25eef8a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:43:30.325963 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:30.325875    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/bfeb358d-808a-4246-a3d4-35add25eef8a-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"bfeb358d-808a-4246-a3d4-35add25eef8a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:43:30.325963 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:30.325917    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/bfeb358d-808a-4246-a3d4-35add25eef8a-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"bfeb358d-808a-4246-a3d4-35add25eef8a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:43:30.325963 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:30.325938    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lgh9m\" (UniqueName: \"kubernetes.io/projected/bfeb358d-808a-4246-a3d4-35add25eef8a-kube-api-access-lgh9m\") pod \"prometheus-k8s-0\" (UID: \"bfeb358d-808a-4246-a3d4-35add25eef8a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:43:30.326271 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:30.325974    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/bfeb358d-808a-4246-a3d4-35add25eef8a-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"bfeb358d-808a-4246-a3d4-35add25eef8a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:43:30.326271 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:30.326010    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bfeb358d-808a-4246-a3d4-35add25eef8a-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"bfeb358d-808a-4246-a3d4-35add25eef8a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:43:30.326271 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:30.326038    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/bfeb358d-808a-4246-a3d4-35add25eef8a-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"bfeb358d-808a-4246-a3d4-35add25eef8a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:43:30.326271 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:30.326076    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/bfeb358d-808a-4246-a3d4-35add25eef8a-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"bfeb358d-808a-4246-a3d4-35add25eef8a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:43:30.326271 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:30.326112    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/bfeb358d-808a-4246-a3d4-35add25eef8a-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"bfeb358d-808a-4246-a3d4-35add25eef8a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:43:30.326271 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:30.326145    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bfeb358d-808a-4246-a3d4-35add25eef8a-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"bfeb358d-808a-4246-a3d4-35add25eef8a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:43:30.326271 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:30.326175    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/bfeb358d-808a-4246-a3d4-35add25eef8a-config\") pod \"prometheus-k8s-0\" (UID: \"bfeb358d-808a-4246-a3d4-35add25eef8a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:43:30.326271 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:30.326205    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bfeb358d-808a-4246-a3d4-35add25eef8a-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"bfeb358d-808a-4246-a3d4-35add25eef8a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:43:30.326271 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:30.326230    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/bfeb358d-808a-4246-a3d4-35add25eef8a-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"bfeb358d-808a-4246-a3d4-35add25eef8a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:43:30.326271 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:30.326264    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/bfeb358d-808a-4246-a3d4-35add25eef8a-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"bfeb358d-808a-4246-a3d4-35add25eef8a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:43:30.326780 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:30.326290    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/bfeb358d-808a-4246-a3d4-35add25eef8a-config-out\") pod \"prometheus-k8s-0\" (UID: \"bfeb358d-808a-4246-a3d4-35add25eef8a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:43:30.326780 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:30.326315    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/bfeb358d-808a-4246-a3d4-35add25eef8a-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"bfeb358d-808a-4246-a3d4-35add25eef8a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:43:30.326780 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:30.326339    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/bfeb358d-808a-4246-a3d4-35add25eef8a-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"bfeb358d-808a-4246-a3d4-35add25eef8a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:43:30.326780 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:30.326507    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/bfeb358d-808a-4246-a3d4-35add25eef8a-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"bfeb358d-808a-4246-a3d4-35add25eef8a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:43:30.326780 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:30.326605    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bfeb358d-808a-4246-a3d4-35add25eef8a-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"bfeb358d-808a-4246-a3d4-35add25eef8a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:43:30.327034 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:30.326914    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/bfeb358d-808a-4246-a3d4-35add25eef8a-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"bfeb358d-808a-4246-a3d4-35add25eef8a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:43:30.327034 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:30.326928    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/bfeb358d-808a-4246-a3d4-35add25eef8a-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"bfeb358d-808a-4246-a3d4-35add25eef8a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:43:30.327595 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:30.327547    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bfeb358d-808a-4246-a3d4-35add25eef8a-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"bfeb358d-808a-4246-a3d4-35add25eef8a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:43:30.327697 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:30.327629    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bfeb358d-808a-4246-a3d4-35add25eef8a-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"bfeb358d-808a-4246-a3d4-35add25eef8a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:43:30.328923 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:30.328896    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/bfeb358d-808a-4246-a3d4-35add25eef8a-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"bfeb358d-808a-4246-a3d4-35add25eef8a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:43:30.329129 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:30.329105    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/bfeb358d-808a-4246-a3d4-35add25eef8a-web-config\") pod \"prometheus-k8s-0\" (UID: \"bfeb358d-808a-4246-a3d4-35add25eef8a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:43:30.329377 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:30.329329    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/bfeb358d-808a-4246-a3d4-35add25eef8a-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"bfeb358d-808a-4246-a3d4-35add25eef8a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:43:30.329841 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:30.329801    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bfeb358d-808a-4246-a3d4-35add25eef8a-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"bfeb358d-808a-4246-a3d4-35add25eef8a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:43:30.330375 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:30.330055    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/bfeb358d-808a-4246-a3d4-35add25eef8a-config-out\") pod \"prometheus-k8s-0\" (UID: \"bfeb358d-808a-4246-a3d4-35add25eef8a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:43:30.330375 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:30.330162    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/bfeb358d-808a-4246-a3d4-35add25eef8a-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"bfeb358d-808a-4246-a3d4-35add25eef8a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:43:30.330375 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:30.330207    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/bfeb358d-808a-4246-a3d4-35add25eef8a-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"bfeb358d-808a-4246-a3d4-35add25eef8a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:43:30.330581 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:30.330417    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/bfeb358d-808a-4246-a3d4-35add25eef8a-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"bfeb358d-808a-4246-a3d4-35add25eef8a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:43:30.330946 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:30.330907    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/bfeb358d-808a-4246-a3d4-35add25eef8a-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"bfeb358d-808a-4246-a3d4-35add25eef8a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:43:30.331570 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:30.331547    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/bfeb358d-808a-4246-a3d4-35add25eef8a-config\") pod \"prometheus-k8s-0\" (UID: \"bfeb358d-808a-4246-a3d4-35add25eef8a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:43:30.331916 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:30.331897    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/bfeb358d-808a-4246-a3d4-35add25eef8a-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"bfeb358d-808a-4246-a3d4-35add25eef8a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:43:30.332226 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:30.332206    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/bfeb358d-808a-4246-a3d4-35add25eef8a-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"bfeb358d-808a-4246-a3d4-35add25eef8a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:43:30.332492 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:30.332473    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/bfeb358d-808a-4246-a3d4-35add25eef8a-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"bfeb358d-808a-4246-a3d4-35add25eef8a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:43:30.341183 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:30.341165    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lgh9m\" (UniqueName: \"kubernetes.io/projected/bfeb358d-808a-4246-a3d4-35add25eef8a-kube-api-access-lgh9m\") pod \"prometheus-k8s-0\" (UID: \"bfeb358d-808a-4246-a3d4-35add25eef8a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:43:30.433019 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:30.432983    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:43:30.718137 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:30.718106    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 17 17:43:30.720084 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:43:30.720060    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbfeb358d_808a_4246_a3d4_35add25eef8a.slice/crio-beb06dbe1576c257a903cd20b19eea048d5393c8bf096f75432c241b13c46f8f WatchSource:0}: Error finding container beb06dbe1576c257a903cd20b19eea048d5393c8bf096f75432c241b13c46f8f: Status 404 returned error can't find the container with id beb06dbe1576c257a903cd20b19eea048d5393c8bf096f75432c241b13c46f8f
Sep 17 17:43:31.064287 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:31.064242    2567 generic.go:356] "Generic (PLEG): container finished" podID="bfeb358d-808a-4246-a3d4-35add25eef8a" containerID="2be913aa2e841d8afa94e45a2526d0af00cc5c9ef0030388a1be9d4abf8e08b6" exitCode=0
Sep 17 17:43:31.064754 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:31.064387    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bfeb358d-808a-4246-a3d4-35add25eef8a","Type":"ContainerDied","Data":"2be913aa2e841d8afa94e45a2526d0af00cc5c9ef0030388a1be9d4abf8e08b6"}
Sep 17 17:43:31.064754 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:31.064506    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bfeb358d-808a-4246-a3d4-35add25eef8a","Type":"ContainerStarted","Data":"beb06dbe1576c257a903cd20b19eea048d5393c8bf096f75432c241b13c46f8f"}
Sep 17 17:43:31.289450 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:31.289410    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8dcb3c07-9ad2-4670-b801-0acce83c6faf" path="/var/lib/kubelet/pods/8dcb3c07-9ad2-4670-b801-0acce83c6faf/volumes"
Sep 17 17:43:32.073237 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:32.073206    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bfeb358d-808a-4246-a3d4-35add25eef8a","Type":"ContainerStarted","Data":"5565681cfa29d01633b8b169c2e69b451ffaf65db57fddb6379e52fb18181075"}
Sep 17 17:43:32.073705 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:32.073281    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bfeb358d-808a-4246-a3d4-35add25eef8a","Type":"ContainerStarted","Data":"34a9008408a5d8738a685a49a7af8a0b1a03fe3ec547741800112470d6b438cc"}
Sep 17 17:43:32.073705 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:32.073331    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bfeb358d-808a-4246-a3d4-35add25eef8a","Type":"ContainerStarted","Data":"32a8500576c08bbb01ad9404b98c97aaf29587ba64c372d684b75c3739feda9d"}
Sep 17 17:43:32.073705 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:32.073376    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bfeb358d-808a-4246-a3d4-35add25eef8a","Type":"ContainerStarted","Data":"49ade802dbaf4bda431fba12d8401f09dd20370cd5afcbbeaa5d8f2276207b61"}
Sep 17 17:43:32.073705 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:32.073419    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bfeb358d-808a-4246-a3d4-35add25eef8a","Type":"ContainerStarted","Data":"0225e457849552ee51ef403b5d14e573b60ea7a291e88ee8700651ad48aa7b6c"}
Sep 17 17:43:32.073705 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:32.073459    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bfeb358d-808a-4246-a3d4-35add25eef8a","Type":"ContainerStarted","Data":"ee8fd83643446b7731f3cc5bfa4ba4ad573e4b531957e130747352c2f73efc46"}
Sep 17 17:43:32.107598 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:32.107517    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.087292226 podStartE2EDuration="2.107495216s" totalImagesPullingTime="0s" totalInitContainerRuntime="20.20299ms" isStatefulPod=false podCreationTimestamp="2026-09-17 17:43:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:43:32.105992203 +0000 UTC m=+153.129597360" watchObservedRunningTime="2026-09-17 17:43:32.107495216 +0000 UTC m=+153.131100352"
Sep 17 17:43:32.394749 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:32.394649    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 17:43:32.492366 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:32.492337    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 17:43:34.396499 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:34.396454    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 17:43:34.519215 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:34.519182    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 17:43:35.433485 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:35.433441    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:43:37.794341 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:37.794303    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7f7c6f4457-2gvv4"
Sep 17 17:43:37.794341 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:37.794350    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7f7c6f4457-2gvv4"
Sep 17 17:43:37.799149 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:37.799124    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7f7c6f4457-2gvv4"
Sep 17 17:43:38.103745 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:38.103678    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7f7c6f4457-2gvv4"
Sep 17 17:43:38.156848 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:38.156820    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-c8d8d9b9c-h5mfl"]
Sep 17 17:43:38.279926 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:38.279895    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 17:43:38.385103 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:38.385033    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 17:43:38.483976 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:38.483944    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 17:43:47.192518 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:47.192441    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 17:43:47.299863 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:47.299830    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 17:43:47.383107 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:47.383073    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 17:43:49.708621 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:43:49.708591    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 17:44:03.180049 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:44:03.180007    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-c8d8d9b9c-h5mfl" podUID="3334a6ee-8f98-4a98-9351-1549e87d813d" containerName="console" containerID="cri-o://4137e52d1e08213c0ab6bde8aa32aeb03b3aeef1a349dc71557cd3b9f644b539" gracePeriod=15
Sep 17 17:44:03.415155 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:44:03.415131    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-c8d8d9b9c-h5mfl"
Sep 17 17:44:03.498120 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:44:03.498038    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3334a6ee-8f98-4a98-9351-1549e87d813d-oauth-serving-cert\") pod \"3334a6ee-8f98-4a98-9351-1549e87d813d\" (UID: \"3334a6ee-8f98-4a98-9351-1549e87d813d\") "
Sep 17 17:44:03.498120 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:44:03.498084    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3334a6ee-8f98-4a98-9351-1549e87d813d-console-config\") pod \"3334a6ee-8f98-4a98-9351-1549e87d813d\" (UID: \"3334a6ee-8f98-4a98-9351-1549e87d813d\") "
Sep 17 17:44:03.498120 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:44:03.498113    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3334a6ee-8f98-4a98-9351-1549e87d813d-trusted-ca-bundle\") pod \"3334a6ee-8f98-4a98-9351-1549e87d813d\" (UID: \"3334a6ee-8f98-4a98-9351-1549e87d813d\") "
Sep 17 17:44:03.498400 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:44:03.498141    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3334a6ee-8f98-4a98-9351-1549e87d813d-console-serving-cert\") pod \"3334a6ee-8f98-4a98-9351-1549e87d813d\" (UID: \"3334a6ee-8f98-4a98-9351-1549e87d813d\") "
Sep 17 17:44:03.498400 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:44:03.498161    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3334a6ee-8f98-4a98-9351-1549e87d813d-console-oauth-config\") pod \"3334a6ee-8f98-4a98-9351-1549e87d813d\" (UID: \"3334a6ee-8f98-4a98-9351-1549e87d813d\") "
Sep 17 17:44:03.498400 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:44:03.498192    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3334a6ee-8f98-4a98-9351-1549e87d813d-service-ca\") pod \"3334a6ee-8f98-4a98-9351-1549e87d813d\" (UID: \"3334a6ee-8f98-4a98-9351-1549e87d813d\") "
Sep 17 17:44:03.498400 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:44:03.498364    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-729d4\" (UniqueName: \"kubernetes.io/projected/3334a6ee-8f98-4a98-9351-1549e87d813d-kube-api-access-729d4\") pod \"3334a6ee-8f98-4a98-9351-1549e87d813d\" (UID: \"3334a6ee-8f98-4a98-9351-1549e87d813d\") "
Sep 17 17:44:03.498625 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:44:03.498523    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3334a6ee-8f98-4a98-9351-1549e87d813d-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "3334a6ee-8f98-4a98-9351-1549e87d813d" (UID: "3334a6ee-8f98-4a98-9351-1549e87d813d"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:44:03.498695 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:44:03.498643    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3334a6ee-8f98-4a98-9351-1549e87d813d-service-ca" (OuterVolumeSpecName: "service-ca") pod "3334a6ee-8f98-4a98-9351-1549e87d813d" (UID: "3334a6ee-8f98-4a98-9351-1549e87d813d"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:44:03.498695 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:44:03.498649    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3334a6ee-8f98-4a98-9351-1549e87d813d-console-config" (OuterVolumeSpecName: "console-config") pod "3334a6ee-8f98-4a98-9351-1549e87d813d" (UID: "3334a6ee-8f98-4a98-9351-1549e87d813d"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:44:03.498695 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:44:03.498653    2567 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3334a6ee-8f98-4a98-9351-1549e87d813d-oauth-serving-cert\") on node \"ip-10-0-139-237.ec2.internal\" DevicePath \"\""
Sep 17 17:44:03.498695 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:44:03.498652    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3334a6ee-8f98-4a98-9351-1549e87d813d-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "3334a6ee-8f98-4a98-9351-1549e87d813d" (UID: "3334a6ee-8f98-4a98-9351-1549e87d813d"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:44:03.500355 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:44:03.500333    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3334a6ee-8f98-4a98-9351-1549e87d813d-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "3334a6ee-8f98-4a98-9351-1549e87d813d" (UID: "3334a6ee-8f98-4a98-9351-1549e87d813d"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:44:03.500498 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:44:03.500361    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3334a6ee-8f98-4a98-9351-1549e87d813d-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "3334a6ee-8f98-4a98-9351-1549e87d813d" (UID: "3334a6ee-8f98-4a98-9351-1549e87d813d"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:44:03.500498 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:44:03.500452    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3334a6ee-8f98-4a98-9351-1549e87d813d-kube-api-access-729d4" (OuterVolumeSpecName: "kube-api-access-729d4") pod "3334a6ee-8f98-4a98-9351-1549e87d813d" (UID: "3334a6ee-8f98-4a98-9351-1549e87d813d"). InnerVolumeSpecName "kube-api-access-729d4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:44:03.600044 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:44:03.600005    2567 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3334a6ee-8f98-4a98-9351-1549e87d813d-console-config\") on node \"ip-10-0-139-237.ec2.internal\" DevicePath \"\""
Sep 17 17:44:03.600044 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:44:03.600044    2567 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3334a6ee-8f98-4a98-9351-1549e87d813d-trusted-ca-bundle\") on node \"ip-10-0-139-237.ec2.internal\" DevicePath \"\""
Sep 17 17:44:03.600245 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:44:03.600059    2567 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3334a6ee-8f98-4a98-9351-1549e87d813d-console-serving-cert\") on node \"ip-10-0-139-237.ec2.internal\" DevicePath \"\""
Sep 17 17:44:03.600245 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:44:03.600072    2567 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3334a6ee-8f98-4a98-9351-1549e87d813d-console-oauth-config\") on node \"ip-10-0-139-237.ec2.internal\" DevicePath \"\""
Sep 17 17:44:03.600245 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:44:03.600084    2567 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3334a6ee-8f98-4a98-9351-1549e87d813d-service-ca\") on node \"ip-10-0-139-237.ec2.internal\" DevicePath \"\""
Sep 17 17:44:03.600245 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:44:03.600097    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-729d4\" (UniqueName: \"kubernetes.io/projected/3334a6ee-8f98-4a98-9351-1549e87d813d-kube-api-access-729d4\") on node \"ip-10-0-139-237.ec2.internal\" DevicePath \"\""
Sep 17 17:44:04.204880 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:44:04.204848    2567 generic.go:356] "Generic (PLEG): container finished" podID="3334a6ee-8f98-4a98-9351-1549e87d813d" containerID="4137e52d1e08213c0ab6bde8aa32aeb03b3aeef1a349dc71557cd3b9f644b539" exitCode=0
Sep 17 17:44:04.205315 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:44:04.204900    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-c8d8d9b9c-h5mfl"
Sep 17 17:44:04.205315 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:44:04.204903    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-c8d8d9b9c-h5mfl" event={"ID":"3334a6ee-8f98-4a98-9351-1549e87d813d","Type":"ContainerDied","Data":"4137e52d1e08213c0ab6bde8aa32aeb03b3aeef1a349dc71557cd3b9f644b539"}
Sep 17 17:44:04.205315 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:44:04.205037    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-c8d8d9b9c-h5mfl" event={"ID":"3334a6ee-8f98-4a98-9351-1549e87d813d","Type":"ContainerDied","Data":"2cbff18c0663ff35243ea09b744aabd02ec2ef1fc571ba0b5fd0bec571a0c82c"}
Sep 17 17:44:04.205315 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:44:04.205058    2567 scope.go:117] "RemoveContainer" containerID="4137e52d1e08213c0ab6bde8aa32aeb03b3aeef1a349dc71557cd3b9f644b539"
Sep 17 17:44:04.213590 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:44:04.213570    2567 scope.go:117] "RemoveContainer" containerID="4137e52d1e08213c0ab6bde8aa32aeb03b3aeef1a349dc71557cd3b9f644b539"
Sep 17 17:44:04.213875 ip-10-0-139-237 kubenswrapper[2567]: E0917 17:44:04.213848    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4137e52d1e08213c0ab6bde8aa32aeb03b3aeef1a349dc71557cd3b9f644b539\": container with ID starting with 4137e52d1e08213c0ab6bde8aa32aeb03b3aeef1a349dc71557cd3b9f644b539 not found: ID does not exist" containerID="4137e52d1e08213c0ab6bde8aa32aeb03b3aeef1a349dc71557cd3b9f644b539"
Sep 17 17:44:04.213919 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:44:04.213882    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4137e52d1e08213c0ab6bde8aa32aeb03b3aeef1a349dc71557cd3b9f644b539"} err="failed to get container status \"4137e52d1e08213c0ab6bde8aa32aeb03b3aeef1a349dc71557cd3b9f644b539\": rpc error: code = NotFound desc = could not find container \"4137e52d1e08213c0ab6bde8aa32aeb03b3aeef1a349dc71557cd3b9f644b539\": container with ID starting with 4137e52d1e08213c0ab6bde8aa32aeb03b3aeef1a349dc71557cd3b9f644b539 not found: ID does not exist"
Sep 17 17:44:04.226847 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:44:04.226825    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-c8d8d9b9c-h5mfl"]
Sep 17 17:44:04.228655 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:44:04.228638    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-c8d8d9b9c-h5mfl"]
Sep 17 17:44:05.285360 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:44:05.285317    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3334a6ee-8f98-4a98-9351-1549e87d813d" path="/var/lib/kubelet/pods/3334a6ee-8f98-4a98-9351-1549e87d813d/volumes"
Sep 17 17:44:17.695080 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:44:17.695040    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 17:44:30.433706 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:44:30.433677    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:44:30.448942 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:44:30.448915    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:44:31.328931 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:44:31.328904    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:44:41.693360 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:44:41.693322    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 17:44:41.760168 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:44:41.760139    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 17:44:54.837544 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:44:54.837510    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 17:44:54.936811 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:44:54.936773    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 17:44:54.997370 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:44:54.997345    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 17:45:17.642426 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:45:17.642347    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 17:45:17.756747 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:45:17.756709    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 17:45:24.445147 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:45:24.445075    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 17:45:24.739960 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:45:24.739873    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 17:45:24.950832 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:45:24.950801    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 17:45:25.050797 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:45:25.050768    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 17:45:25.355609 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:45:25.355514    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 17:45:25.593836 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:45:25.593780    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 17:45:25.685922 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:45:25.685835    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 17:45:25.890041 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:45:25.890011    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 17:45:26.169155 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:45:26.169124    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 17:45:26.374760 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:45:26.374723    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 17:45:26.475389 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:45:26.475299    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 17:46:01.212798 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:46:01.212771    2567 kubelet.go:1648] "Image garbage collection succeeded"
Sep 17 17:46:16.335065 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:46:16.335028    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 17:46:17.609711 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:46:17.609683    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 17:46:44.861439 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:46:44.861360    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 17:46:45.102000 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:46:45.101961    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 17:46:45.163075 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:46:45.162998    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-64568bcd49-8h4vd"]
Sep 17 17:46:45.163358 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:46:45.163342    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3334a6ee-8f98-4a98-9351-1549e87d813d" containerName="console"
Sep 17 17:46:45.163444 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:46:45.163362    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="3334a6ee-8f98-4a98-9351-1549e87d813d" containerName="console"
Sep 17 17:46:45.163495 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:46:45.163456    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="3334a6ee-8f98-4a98-9351-1549e87d813d" containerName="console"
Sep 17 17:46:45.166107 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:46:45.166087    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-8h4vd"
Sep 17 17:46:45.178449 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:46:45.178430    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Sep 17 17:46:45.179459 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:46:45.179442    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Sep 17 17:46:45.179568 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:46:45.179442    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-4qj7d\""
Sep 17 17:46:45.182608 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:46:45.182593    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Sep 17 17:46:45.190700 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:46:45.190681    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-8h4vd"]
Sep 17 17:46:45.211040 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:46:45.211016    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/ca127cca-a738-466b-a344-60e2f0943ae7-data\") pod \"seaweedfs-64568bcd49-8h4vd\" (UID: \"ca127cca-a738-466b-a344-60e2f0943ae7\") " pod="kserve/seaweedfs-64568bcd49-8h4vd"
Sep 17 17:46:45.211124 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:46:45.211066    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fll2p\" (UniqueName: \"kubernetes.io/projected/ca127cca-a738-466b-a344-60e2f0943ae7-kube-api-access-fll2p\") pod \"seaweedfs-64568bcd49-8h4vd\" (UID: \"ca127cca-a738-466b-a344-60e2f0943ae7\") " pod="kserve/seaweedfs-64568bcd49-8h4vd"
Sep 17 17:46:45.304815 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:46:45.304770    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 17:46:45.312182 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:46:45.312157    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fll2p\" (UniqueName: \"kubernetes.io/projected/ca127cca-a738-466b-a344-60e2f0943ae7-kube-api-access-fll2p\") pod \"seaweedfs-64568bcd49-8h4vd\" (UID: \"ca127cca-a738-466b-a344-60e2f0943ae7\") " pod="kserve/seaweedfs-64568bcd49-8h4vd"
Sep 17 17:46:45.312286 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:46:45.312236    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/ca127cca-a738-466b-a344-60e2f0943ae7-data\") pod \"seaweedfs-64568bcd49-8h4vd\" (UID: \"ca127cca-a738-466b-a344-60e2f0943ae7\") " pod="kserve/seaweedfs-64568bcd49-8h4vd"
Sep 17 17:46:45.312617 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:46:45.312598    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/ca127cca-a738-466b-a344-60e2f0943ae7-data\") pod \"seaweedfs-64568bcd49-8h4vd\" (UID: \"ca127cca-a738-466b-a344-60e2f0943ae7\") " pod="kserve/seaweedfs-64568bcd49-8h4vd"
Sep 17 17:46:45.323601 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:46:45.323581    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fll2p\" (UniqueName: \"kubernetes.io/projected/ca127cca-a738-466b-a344-60e2f0943ae7-kube-api-access-fll2p\") pod \"seaweedfs-64568bcd49-8h4vd\" (UID: \"ca127cca-a738-466b-a344-60e2f0943ae7\") " pod="kserve/seaweedfs-64568bcd49-8h4vd"
Sep 17 17:46:45.473903 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:46:45.473819    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 17:46:45.479768 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:46:45.479750    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-8h4vd"
Sep 17 17:46:45.761879 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:46:45.761844    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-8h4vd"]
Sep 17 17:46:45.763872 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:46:45.763846    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca127cca_a738_466b_a344_60e2f0943ae7.slice/crio-87415e49e7a162e7f2ae7cee345e356433d698c22c96fdd80075a5215e5fb006 WatchSource:0}: Error finding container 87415e49e7a162e7f2ae7cee345e356433d698c22c96fdd80075a5215e5fb006: Status 404 returned error can't find the container with id 87415e49e7a162e7f2ae7cee345e356433d698c22c96fdd80075a5215e5fb006
Sep 17 17:46:45.765260 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:46:45.765239    2567 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 17:46:45.793159 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:46:45.793135    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-8h4vd" event={"ID":"ca127cca-a738-466b-a344-60e2f0943ae7","Type":"ContainerStarted","Data":"87415e49e7a162e7f2ae7cee345e356433d698c22c96fdd80075a5215e5fb006"}
Sep 17 17:46:46.304297 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:46:46.304262    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 17:46:46.407502 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:46:46.407469    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 17:46:49.810509 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:46:49.810477    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-8h4vd" event={"ID":"ca127cca-a738-466b-a344-60e2f0943ae7","Type":"ContainerStarted","Data":"9302effc0c685b4799b391fc53eff1383385054f710b1849319590c9fd94ac13"}
Sep 17 17:46:49.810863 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:46:49.810608    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/seaweedfs-64568bcd49-8h4vd"
Sep 17 17:46:49.830220 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:46:49.830177    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-64568bcd49-8h4vd" podStartSLOduration=1.725710518 podStartE2EDuration="4.830162531s" totalImagesPullingTime="3.104452013s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:46:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:46:49.828910405 +0000 UTC m=+350.852515572" watchObservedRunningTime="2026-09-17 17:46:49.830162531 +0000 UTC m=+350.853767667"
Sep 17 17:46:55.817095 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:46:55.817065    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/seaweedfs-64568bcd49-8h4vd"
Sep 17 17:47:01.633871 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:47:01.633827    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5f968588f4-6rgtl"]
Sep 17 17:47:01.638522 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:47:01.638500    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5f968588f4-6rgtl"
Sep 17 17:47:01.650991 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:47:01.650966    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5f968588f4-6rgtl"]
Sep 17 17:47:01.694288 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:47:01.694257    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 17:47:01.751146 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:47:01.751118    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/cca22102-4b4b-46f7-8f37-fdced486b623-console-serving-cert\") pod \"console-5f968588f4-6rgtl\" (UID: \"cca22102-4b4b-46f7-8f37-fdced486b623\") " pod="openshift-console/console-5f968588f4-6rgtl"
Sep 17 17:47:01.751270 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:47:01.751163    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/cca22102-4b4b-46f7-8f37-fdced486b623-console-config\") pod \"console-5f968588f4-6rgtl\" (UID: \"cca22102-4b4b-46f7-8f37-fdced486b623\") " pod="openshift-console/console-5f968588f4-6rgtl"
Sep 17 17:47:01.751270 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:47:01.751229    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2p5fq\" (UniqueName: \"kubernetes.io/projected/cca22102-4b4b-46f7-8f37-fdced486b623-kube-api-access-2p5fq\") pod \"console-5f968588f4-6rgtl\" (UID: \"cca22102-4b4b-46f7-8f37-fdced486b623\") " pod="openshift-console/console-5f968588f4-6rgtl"
Sep 17 17:47:01.751270 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:47:01.751255    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/cca22102-4b4b-46f7-8f37-fdced486b623-oauth-serving-cert\") pod \"console-5f968588f4-6rgtl\" (UID: \"cca22102-4b4b-46f7-8f37-fdced486b623\") " pod="openshift-console/console-5f968588f4-6rgtl"
Sep 17 17:47:01.751390 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:47:01.751282    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/cca22102-4b4b-46f7-8f37-fdced486b623-service-ca\") pod \"console-5f968588f4-6rgtl\" (UID: \"cca22102-4b4b-46f7-8f37-fdced486b623\") " pod="openshift-console/console-5f968588f4-6rgtl"
Sep 17 17:47:01.751390 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:47:01.751300    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cca22102-4b4b-46f7-8f37-fdced486b623-trusted-ca-bundle\") pod \"console-5f968588f4-6rgtl\" (UID: \"cca22102-4b4b-46f7-8f37-fdced486b623\") " pod="openshift-console/console-5f968588f4-6rgtl"
Sep 17 17:47:01.751390 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:47:01.751327    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/cca22102-4b4b-46f7-8f37-fdced486b623-console-oauth-config\") pod \"console-5f968588f4-6rgtl\" (UID: \"cca22102-4b4b-46f7-8f37-fdced486b623\") " pod="openshift-console/console-5f968588f4-6rgtl"
Sep 17 17:47:01.794145 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:47:01.794119    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 17:47:01.852080 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:47:01.852047    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/cca22102-4b4b-46f7-8f37-fdced486b623-console-config\") pod \"console-5f968588f4-6rgtl\" (UID: \"cca22102-4b4b-46f7-8f37-fdced486b623\") " pod="openshift-console/console-5f968588f4-6rgtl"
Sep 17 17:47:01.852248 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:47:01.852112    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2p5fq\" (UniqueName: \"kubernetes.io/projected/cca22102-4b4b-46f7-8f37-fdced486b623-kube-api-access-2p5fq\") pod \"console-5f968588f4-6rgtl\" (UID: \"cca22102-4b4b-46f7-8f37-fdced486b623\") " pod="openshift-console/console-5f968588f4-6rgtl"
Sep 17 17:47:01.852248 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:47:01.852145    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/cca22102-4b4b-46f7-8f37-fdced486b623-oauth-serving-cert\") pod \"console-5f968588f4-6rgtl\" (UID: \"cca22102-4b4b-46f7-8f37-fdced486b623\") " pod="openshift-console/console-5f968588f4-6rgtl"
Sep 17 17:47:01.852248 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:47:01.852176    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/cca22102-4b4b-46f7-8f37-fdced486b623-service-ca\") pod \"console-5f968588f4-6rgtl\" (UID: \"cca22102-4b4b-46f7-8f37-fdced486b623\") " pod="openshift-console/console-5f968588f4-6rgtl"
Sep 17 17:47:01.852248 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:47:01.852203    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cca22102-4b4b-46f7-8f37-fdced486b623-trusted-ca-bundle\") pod \"console-5f968588f4-6rgtl\" (UID: \"cca22102-4b4b-46f7-8f37-fdced486b623\") " pod="openshift-console/console-5f968588f4-6rgtl"
Sep 17 17:47:01.852248 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:47:01.852237    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/cca22102-4b4b-46f7-8f37-fdced486b623-console-oauth-config\") pod \"console-5f968588f4-6rgtl\" (UID: \"cca22102-4b4b-46f7-8f37-fdced486b623\") " pod="openshift-console/console-5f968588f4-6rgtl"
Sep 17 17:47:01.852506 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:47:01.852267    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/cca22102-4b4b-46f7-8f37-fdced486b623-console-serving-cert\") pod \"console-5f968588f4-6rgtl\" (UID: \"cca22102-4b4b-46f7-8f37-fdced486b623\") " pod="openshift-console/console-5f968588f4-6rgtl"
Sep 17 17:47:01.852913 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:47:01.852888    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/cca22102-4b4b-46f7-8f37-fdced486b623-console-config\") pod \"console-5f968588f4-6rgtl\" (UID: \"cca22102-4b4b-46f7-8f37-fdced486b623\") " pod="openshift-console/console-5f968588f4-6rgtl"
Sep 17 17:47:01.853037 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:47:01.852931    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/cca22102-4b4b-46f7-8f37-fdced486b623-oauth-serving-cert\") pod \"console-5f968588f4-6rgtl\" (UID: \"cca22102-4b4b-46f7-8f37-fdced486b623\") " pod="openshift-console/console-5f968588f4-6rgtl"
Sep 17 17:47:01.853037 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:47:01.852958    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/cca22102-4b4b-46f7-8f37-fdced486b623-service-ca\") pod \"console-5f968588f4-6rgtl\" (UID: \"cca22102-4b4b-46f7-8f37-fdced486b623\") " pod="openshift-console/console-5f968588f4-6rgtl"
Sep 17 17:47:01.853120 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:47:01.853099    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cca22102-4b4b-46f7-8f37-fdced486b623-trusted-ca-bundle\") pod \"console-5f968588f4-6rgtl\" (UID: \"cca22102-4b4b-46f7-8f37-fdced486b623\") " pod="openshift-console/console-5f968588f4-6rgtl"
Sep 17 17:47:01.855297 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:47:01.855274    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/cca22102-4b4b-46f7-8f37-fdced486b623-console-oauth-config\") pod \"console-5f968588f4-6rgtl\" (UID: \"cca22102-4b4b-46f7-8f37-fdced486b623\") " pod="openshift-console/console-5f968588f4-6rgtl"
Sep 17 17:47:01.855405 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:47:01.855342    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/cca22102-4b4b-46f7-8f37-fdced486b623-console-serving-cert\") pod \"console-5f968588f4-6rgtl\" (UID: \"cca22102-4b4b-46f7-8f37-fdced486b623\") " pod="openshift-console/console-5f968588f4-6rgtl"
Sep 17 17:47:01.860765 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:47:01.860744    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2p5fq\" (UniqueName: \"kubernetes.io/projected/cca22102-4b4b-46f7-8f37-fdced486b623-kube-api-access-2p5fq\") pod \"console-5f968588f4-6rgtl\" (UID: \"cca22102-4b4b-46f7-8f37-fdced486b623\") " pod="openshift-console/console-5f968588f4-6rgtl"
Sep 17 17:47:01.894138 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:47:01.894078    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 17:47:01.948252 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:47:01.948229    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5f968588f4-6rgtl"
Sep 17 17:47:02.238387 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:47:02.238317    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5f968588f4-6rgtl"]
Sep 17 17:47:02.241593 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:47:02.241562    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcca22102_4b4b_46f7_8f37_fdced486b623.slice/crio-201c28813e3a0a18104b6cdc8e372d6f6c9b1d503b0c54ed5653b8fdf9595bec WatchSource:0}: Error finding container 201c28813e3a0a18104b6cdc8e372d6f6c9b1d503b0c54ed5653b8fdf9595bec: Status 404 returned error can't find the container with id 201c28813e3a0a18104b6cdc8e372d6f6c9b1d503b0c54ed5653b8fdf9595bec
Sep 17 17:47:02.439175 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:47:02.439143    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 17:47:02.540155 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:47:02.540126    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 17:47:02.862045 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:47:02.861966    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5f968588f4-6rgtl" event={"ID":"cca22102-4b4b-46f7-8f37-fdced486b623","Type":"ContainerStarted","Data":"83baa60c90fd9c7f2889bbaf6a176927592f1cb51de563e8b8e7bf5f74cbb6a9"}
Sep 17 17:47:02.862045 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:47:02.862011    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5f968588f4-6rgtl" event={"ID":"cca22102-4b4b-46f7-8f37-fdced486b623","Type":"ContainerStarted","Data":"201c28813e3a0a18104b6cdc8e372d6f6c9b1d503b0c54ed5653b8fdf9595bec"}
Sep 17 17:47:02.897342 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:47:02.897294    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5f968588f4-6rgtl" podStartSLOduration=1.897279756 podStartE2EDuration="1.897279756s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:47:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:47:02.896640187 +0000 UTC m=+363.920245324" watchObservedRunningTime="2026-09-17 17:47:02.897279756 +0000 UTC m=+363.920884893"
Sep 17 17:47:09.851151 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:47:09.851121    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 17:47:11.949200 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:47:11.949169    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5f968588f4-6rgtl"
Sep 17 17:47:11.949681 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:47:11.949455    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5f968588f4-6rgtl"
Sep 17 17:47:11.953971 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:47:11.953946    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5f968588f4-6rgtl"
Sep 17 17:47:12.903737 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:47:12.903708    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5f968588f4-6rgtl"
Sep 17 17:47:12.952999 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:47:12.952965    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7f7c6f4457-2gvv4"]
Sep 17 17:47:13.090053 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:47:13.090023    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 17:47:13.190520 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:47:13.190445    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 17:47:13.290965 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:47:13.290933    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 17:47:37.971997 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:47:37.971958    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-7f7c6f4457-2gvv4" podUID="d1d7e2c1-4706-45de-87fa-7d8d7c7ddfe4" containerName="console" containerID="cri-o://12e3cfe49742d0ee1bf664654b4a5b44f4f5935f2516f3e1177af2acd859aa11" gracePeriod=15
Sep 17 17:47:38.100997 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:47:38.100958    2567 patch_prober.go:28] interesting pod/console-7f7c6f4457-2gvv4 container/console namespace/openshift-console: Readiness probe status=failure output="Get \"https://10.134.0.19:8443/health\": dial tcp 10.134.0.19:8443: connect: connection refused" start-of-body=
Sep 17 17:47:38.101114 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:47:38.101037    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-console/console-7f7c6f4457-2gvv4" podUID="d1d7e2c1-4706-45de-87fa-7d8d7c7ddfe4" containerName="console" probeResult="failure" output="Get \"https://10.134.0.19:8443/health\": dial tcp 10.134.0.19:8443: connect: connection refused"
Sep 17 17:47:38.208474 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:47:38.208450    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7f7c6f4457-2gvv4"
Sep 17 17:47:38.243853 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:47:38.243789    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bn24n\" (UniqueName: \"kubernetes.io/projected/d1d7e2c1-4706-45de-87fa-7d8d7c7ddfe4-kube-api-access-bn24n\") pod \"d1d7e2c1-4706-45de-87fa-7d8d7c7ddfe4\" (UID: \"d1d7e2c1-4706-45de-87fa-7d8d7c7ddfe4\") "
Sep 17 17:47:38.243853 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:47:38.243825    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d1d7e2c1-4706-45de-87fa-7d8d7c7ddfe4-oauth-serving-cert\") pod \"d1d7e2c1-4706-45de-87fa-7d8d7c7ddfe4\" (UID: \"d1d7e2c1-4706-45de-87fa-7d8d7c7ddfe4\") "
Sep 17 17:47:38.243853 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:47:38.243842    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d1d7e2c1-4706-45de-87fa-7d8d7c7ddfe4-service-ca\") pod \"d1d7e2c1-4706-45de-87fa-7d8d7c7ddfe4\" (UID: \"d1d7e2c1-4706-45de-87fa-7d8d7c7ddfe4\") "
Sep 17 17:47:38.244088 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:47:38.243873    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d1d7e2c1-4706-45de-87fa-7d8d7c7ddfe4-trusted-ca-bundle\") pod \"d1d7e2c1-4706-45de-87fa-7d8d7c7ddfe4\" (UID: \"d1d7e2c1-4706-45de-87fa-7d8d7c7ddfe4\") "
Sep 17 17:47:38.244088 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:47:38.243894    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d1d7e2c1-4706-45de-87fa-7d8d7c7ddfe4-console-config\") pod \"d1d7e2c1-4706-45de-87fa-7d8d7c7ddfe4\" (UID: \"d1d7e2c1-4706-45de-87fa-7d8d7c7ddfe4\") "
Sep 17 17:47:38.244088 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:47:38.243923    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d1d7e2c1-4706-45de-87fa-7d8d7c7ddfe4-console-oauth-config\") pod \"d1d7e2c1-4706-45de-87fa-7d8d7c7ddfe4\" (UID: \"d1d7e2c1-4706-45de-87fa-7d8d7c7ddfe4\") "
Sep 17 17:47:38.244088 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:47:38.243949    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d1d7e2c1-4706-45de-87fa-7d8d7c7ddfe4-console-serving-cert\") pod \"d1d7e2c1-4706-45de-87fa-7d8d7c7ddfe4\" (UID: \"d1d7e2c1-4706-45de-87fa-7d8d7c7ddfe4\") "
Sep 17 17:47:38.244287 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:47:38.244241    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1d7e2c1-4706-45de-87fa-7d8d7c7ddfe4-service-ca" (OuterVolumeSpecName: "service-ca") pod "d1d7e2c1-4706-45de-87fa-7d8d7c7ddfe4" (UID: "d1d7e2c1-4706-45de-87fa-7d8d7c7ddfe4"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:47:38.244359 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:47:38.244331    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1d7e2c1-4706-45de-87fa-7d8d7c7ddfe4-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "d1d7e2c1-4706-45de-87fa-7d8d7c7ddfe4" (UID: "d1d7e2c1-4706-45de-87fa-7d8d7c7ddfe4"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:47:38.244423 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:47:38.244340    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1d7e2c1-4706-45de-87fa-7d8d7c7ddfe4-console-config" (OuterVolumeSpecName: "console-config") pod "d1d7e2c1-4706-45de-87fa-7d8d7c7ddfe4" (UID: "d1d7e2c1-4706-45de-87fa-7d8d7c7ddfe4"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:47:38.244696 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:47:38.244674    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1d7e2c1-4706-45de-87fa-7d8d7c7ddfe4-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "d1d7e2c1-4706-45de-87fa-7d8d7c7ddfe4" (UID: "d1d7e2c1-4706-45de-87fa-7d8d7c7ddfe4"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:47:38.246244 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:47:38.246220    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d1d7e2c1-4706-45de-87fa-7d8d7c7ddfe4-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "d1d7e2c1-4706-45de-87fa-7d8d7c7ddfe4" (UID: "d1d7e2c1-4706-45de-87fa-7d8d7c7ddfe4"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:47:38.246429 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:47:38.246408    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1d7e2c1-4706-45de-87fa-7d8d7c7ddfe4-kube-api-access-bn24n" (OuterVolumeSpecName: "kube-api-access-bn24n") pod "d1d7e2c1-4706-45de-87fa-7d8d7c7ddfe4" (UID: "d1d7e2c1-4706-45de-87fa-7d8d7c7ddfe4"). InnerVolumeSpecName "kube-api-access-bn24n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:47:38.246492 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:47:38.246412    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d1d7e2c1-4706-45de-87fa-7d8d7c7ddfe4-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "d1d7e2c1-4706-45de-87fa-7d8d7c7ddfe4" (UID: "d1d7e2c1-4706-45de-87fa-7d8d7c7ddfe4"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:47:38.344625 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:47:38.344591    2567 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d1d7e2c1-4706-45de-87fa-7d8d7c7ddfe4-console-oauth-config\") on node \"ip-10-0-139-237.ec2.internal\" DevicePath \"\""
Sep 17 17:47:38.344625 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:47:38.344624    2567 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d1d7e2c1-4706-45de-87fa-7d8d7c7ddfe4-console-serving-cert\") on node \"ip-10-0-139-237.ec2.internal\" DevicePath \"\""
Sep 17 17:47:38.344871 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:47:38.344639    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bn24n\" (UniqueName: \"kubernetes.io/projected/d1d7e2c1-4706-45de-87fa-7d8d7c7ddfe4-kube-api-access-bn24n\") on node \"ip-10-0-139-237.ec2.internal\" DevicePath \"\""
Sep 17 17:47:38.344871 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:47:38.344653    2567 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d1d7e2c1-4706-45de-87fa-7d8d7c7ddfe4-oauth-serving-cert\") on node \"ip-10-0-139-237.ec2.internal\" DevicePath \"\""
Sep 17 17:47:38.344871 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:47:38.344667    2567 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d1d7e2c1-4706-45de-87fa-7d8d7c7ddfe4-service-ca\") on node \"ip-10-0-139-237.ec2.internal\" DevicePath \"\""
Sep 17 17:47:38.344871 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:47:38.344680    2567 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d1d7e2c1-4706-45de-87fa-7d8d7c7ddfe4-trusted-ca-bundle\") on node \"ip-10-0-139-237.ec2.internal\" DevicePath \"\""
Sep 17 17:47:38.344871 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:47:38.344692    2567 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d1d7e2c1-4706-45de-87fa-7d8d7c7ddfe4-console-config\") on node \"ip-10-0-139-237.ec2.internal\" DevicePath \"\""
Sep 17 17:47:38.997378 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:47:38.997346    2567 generic.go:356] "Generic (PLEG): container finished" podID="d1d7e2c1-4706-45de-87fa-7d8d7c7ddfe4" containerID="12e3cfe49742d0ee1bf664654b4a5b44f4f5935f2516f3e1177af2acd859aa11" exitCode=0
Sep 17 17:47:38.997821 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:47:38.997390    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7f7c6f4457-2gvv4"
Sep 17 17:47:38.997821 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:47:38.997403    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7f7c6f4457-2gvv4" event={"ID":"d1d7e2c1-4706-45de-87fa-7d8d7c7ddfe4","Type":"ContainerDied","Data":"12e3cfe49742d0ee1bf664654b4a5b44f4f5935f2516f3e1177af2acd859aa11"}
Sep 17 17:47:38.997821 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:47:38.997442    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7f7c6f4457-2gvv4" event={"ID":"d1d7e2c1-4706-45de-87fa-7d8d7c7ddfe4","Type":"ContainerDied","Data":"195a47adf75b77d14e5c8f26fc660c4eb46c2bf6de0c6ed8c5299fae202359ad"}
Sep 17 17:47:38.997821 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:47:38.997461    2567 scope.go:117] "RemoveContainer" containerID="12e3cfe49742d0ee1bf664654b4a5b44f4f5935f2516f3e1177af2acd859aa11"
Sep 17 17:47:39.005899 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:47:39.005883    2567 scope.go:117] "RemoveContainer" containerID="12e3cfe49742d0ee1bf664654b4a5b44f4f5935f2516f3e1177af2acd859aa11"
Sep 17 17:47:39.006147 ip-10-0-139-237 kubenswrapper[2567]: E0917 17:47:39.006127    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"12e3cfe49742d0ee1bf664654b4a5b44f4f5935f2516f3e1177af2acd859aa11\": container with ID starting with 12e3cfe49742d0ee1bf664654b4a5b44f4f5935f2516f3e1177af2acd859aa11 not found: ID does not exist" containerID="12e3cfe49742d0ee1bf664654b4a5b44f4f5935f2516f3e1177af2acd859aa11"
Sep 17 17:47:39.006196 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:47:39.006155    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"12e3cfe49742d0ee1bf664654b4a5b44f4f5935f2516f3e1177af2acd859aa11"} err="failed to get container status \"12e3cfe49742d0ee1bf664654b4a5b44f4f5935f2516f3e1177af2acd859aa11\": rpc error: code = NotFound desc = could not find container \"12e3cfe49742d0ee1bf664654b4a5b44f4f5935f2516f3e1177af2acd859aa11\": container with ID starting with 12e3cfe49742d0ee1bf664654b4a5b44f4f5935f2516f3e1177af2acd859aa11 not found: ID does not exist"
Sep 17 17:47:39.019202 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:47:39.019182    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7f7c6f4457-2gvv4"]
Sep 17 17:47:39.025922 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:47:39.025901    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7f7c6f4457-2gvv4"]
Sep 17 17:47:39.286082 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:47:39.286003    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d1d7e2c1-4706-45de-87fa-7d8d7c7ddfe4" path="/var/lib/kubelet/pods/d1d7e2c1-4706-45de-87fa-7d8d7c7ddfe4/volumes"
Sep 17 17:48:03.503732 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:48:03.503700    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 17:48:03.576825 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:48:03.576790    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/model-serving-api-7b954ffc95-lt5tg"]
Sep 17 17:48:03.577130 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:48:03.577117    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d1d7e2c1-4706-45de-87fa-7d8d7c7ddfe4" containerName="console"
Sep 17 17:48:03.577177 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:48:03.577132    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="d1d7e2c1-4706-45de-87fa-7d8d7c7ddfe4" containerName="console"
Sep 17 17:48:03.577211 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:48:03.577201    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="d1d7e2c1-4706-45de-87fa-7d8d7c7ddfe4" containerName="console"
Sep 17 17:48:03.579404 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:48:03.579389    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-7b954ffc95-lt5tg"
Sep 17 17:48:03.597871 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:48:03.597843    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-tls\""
Sep 17 17:48:03.597976 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:48:03.597896    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-dockercfg-t7j2p\""
Sep 17 17:48:03.627916 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:48:03.627890    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-7b954ffc95-lt5tg"]
Sep 17 17:48:03.643578 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:48:03.643551    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k2wz7\" (UniqueName: \"kubernetes.io/projected/00a7cd51-8a31-4d58-87f2-16280e5e0774-kube-api-access-k2wz7\") pod \"model-serving-api-7b954ffc95-lt5tg\" (UID: \"00a7cd51-8a31-4d58-87f2-16280e5e0774\") " pod="kserve/model-serving-api-7b954ffc95-lt5tg"
Sep 17 17:48:03.643714 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:48:03.643590    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/00a7cd51-8a31-4d58-87f2-16280e5e0774-tls-certs\") pod \"model-serving-api-7b954ffc95-lt5tg\" (UID: \"00a7cd51-8a31-4d58-87f2-16280e5e0774\") " pod="kserve/model-serving-api-7b954ffc95-lt5tg"
Sep 17 17:48:03.697116 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:48:03.697086    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 17:48:03.744865 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:48:03.744840    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2wz7\" (UniqueName: \"kubernetes.io/projected/00a7cd51-8a31-4d58-87f2-16280e5e0774-kube-api-access-k2wz7\") pod \"model-serving-api-7b954ffc95-lt5tg\" (UID: \"00a7cd51-8a31-4d58-87f2-16280e5e0774\") " pod="kserve/model-serving-api-7b954ffc95-lt5tg"
Sep 17 17:48:03.744984 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:48:03.744873    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/00a7cd51-8a31-4d58-87f2-16280e5e0774-tls-certs\") pod \"model-serving-api-7b954ffc95-lt5tg\" (UID: \"00a7cd51-8a31-4d58-87f2-16280e5e0774\") " pod="kserve/model-serving-api-7b954ffc95-lt5tg"
Sep 17 17:48:03.744984 ip-10-0-139-237 kubenswrapper[2567]: E0917 17:48:03.744977    2567 secret.go:189] Couldn't get secret kserve/model-serving-api-tls: secret "model-serving-api-tls" not found
Sep 17 17:48:03.745063 ip-10-0-139-237 kubenswrapper[2567]: E0917 17:48:03.745032    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/00a7cd51-8a31-4d58-87f2-16280e5e0774-tls-certs podName:00a7cd51-8a31-4d58-87f2-16280e5e0774 nodeName:}" failed. No retries permitted until 2026-09-17 17:48:04.245015448 +0000 UTC m=+425.268620762 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/00a7cd51-8a31-4d58-87f2-16280e5e0774-tls-certs") pod "model-serving-api-7b954ffc95-lt5tg" (UID: "00a7cd51-8a31-4d58-87f2-16280e5e0774") : secret "model-serving-api-tls" not found
Sep 17 17:48:03.765360 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:48:03.765300    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k2wz7\" (UniqueName: \"kubernetes.io/projected/00a7cd51-8a31-4d58-87f2-16280e5e0774-kube-api-access-k2wz7\") pod \"model-serving-api-7b954ffc95-lt5tg\" (UID: \"00a7cd51-8a31-4d58-87f2-16280e5e0774\") " pod="kserve/model-serving-api-7b954ffc95-lt5tg"
Sep 17 17:48:03.794710 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:48:03.794677    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 17:48:04.250298 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:48:04.250258    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/00a7cd51-8a31-4d58-87f2-16280e5e0774-tls-certs\") pod \"model-serving-api-7b954ffc95-lt5tg\" (UID: \"00a7cd51-8a31-4d58-87f2-16280e5e0774\") " pod="kserve/model-serving-api-7b954ffc95-lt5tg"
Sep 17 17:48:04.253453 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:48:04.253427    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/00a7cd51-8a31-4d58-87f2-16280e5e0774-tls-certs\") pod \"model-serving-api-7b954ffc95-lt5tg\" (UID: \"00a7cd51-8a31-4d58-87f2-16280e5e0774\") " pod="kserve/model-serving-api-7b954ffc95-lt5tg"
Sep 17 17:48:04.321262 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:48:04.321231    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 17:48:04.488315 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:48:04.488282    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-7b954ffc95-lt5tg"
Sep 17 17:48:04.764682 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:48:04.764657    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-7b954ffc95-lt5tg"]
Sep 17 17:48:04.766681 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:48:04.766651    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00a7cd51_8a31_4d58_87f2_16280e5e0774.slice/crio-a63dc5345bb02f7bac71454bbfa9d04415c5a1faa5f09eee620c1959528ec907 WatchSource:0}: Error finding container a63dc5345bb02f7bac71454bbfa9d04415c5a1faa5f09eee620c1959528ec907: Status 404 returned error can't find the container with id a63dc5345bb02f7bac71454bbfa9d04415c5a1faa5f09eee620c1959528ec907
Sep 17 17:48:05.095769 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:48:05.095734    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-7b954ffc95-lt5tg" event={"ID":"00a7cd51-8a31-4d58-87f2-16280e5e0774","Type":"ContainerStarted","Data":"a63dc5345bb02f7bac71454bbfa9d04415c5a1faa5f09eee620c1959528ec907"}
Sep 17 17:48:05.464990 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:48:05.464906    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 17:48:05.531781 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:48:05.531736    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 17:48:07.105494 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:48:07.105463    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-7b954ffc95-lt5tg" event={"ID":"00a7cd51-8a31-4d58-87f2-16280e5e0774","Type":"ContainerStarted","Data":"4df1fc388272f05eb0fd6a1cb441ece4abe15cf3d721da8fce958dec6191a0f2"}
Sep 17 17:48:07.105840 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:48:07.105525    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/model-serving-api-7b954ffc95-lt5tg"
Sep 17 17:48:07.124037 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:48:07.124000    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/model-serving-api-7b954ffc95-lt5tg" podStartSLOduration=1.961258678 podStartE2EDuration="4.123985761s" totalImagesPullingTime="2.162727083s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:48:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:48:07.122654976 +0000 UTC m=+428.146260119" watchObservedRunningTime="2026-09-17 17:48:07.123985761 +0000 UTC m=+428.147590897"
Sep 17 17:48:18.113934 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:48:18.113861    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/model-serving-api-7b954ffc95-lt5tg"
Sep 17 17:48:19.046008 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:48:19.043746    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-init-djfmf"]
Sep 17 17:48:19.047294 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:48:19.047274    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-djfmf"
Sep 17 17:48:19.050167 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:48:19.050151    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"hf-token\""
Sep 17 17:48:19.055786 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:48:19.055765    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-djfmf"]
Sep 17 17:48:19.176368 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:48:19.176338    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4frmm\" (UniqueName: \"kubernetes.io/projected/eb97e72b-2549-4c22-889e-bacf215d6b26-kube-api-access-4frmm\") pod \"s3-init-djfmf\" (UID: \"eb97e72b-2549-4c22-889e-bacf215d6b26\") " pod="kserve/s3-init-djfmf"
Sep 17 17:48:19.176752 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:48:19.176379    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eb97e72b-2549-4c22-889e-bacf215d6b26-model-cache\") pod \"s3-init-djfmf\" (UID: \"eb97e72b-2549-4c22-889e-bacf215d6b26\") " pod="kserve/s3-init-djfmf"
Sep 17 17:48:19.277707 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:48:19.277672    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4frmm\" (UniqueName: \"kubernetes.io/projected/eb97e72b-2549-4c22-889e-bacf215d6b26-kube-api-access-4frmm\") pod \"s3-init-djfmf\" (UID: \"eb97e72b-2549-4c22-889e-bacf215d6b26\") " pod="kserve/s3-init-djfmf"
Sep 17 17:48:19.277870 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:48:19.277719    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eb97e72b-2549-4c22-889e-bacf215d6b26-model-cache\") pod \"s3-init-djfmf\" (UID: \"eb97e72b-2549-4c22-889e-bacf215d6b26\") " pod="kserve/s3-init-djfmf"
Sep 17 17:48:19.278140 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:48:19.278122    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eb97e72b-2549-4c22-889e-bacf215d6b26-model-cache\") pod \"s3-init-djfmf\" (UID: \"eb97e72b-2549-4c22-889e-bacf215d6b26\") " pod="kserve/s3-init-djfmf"
Sep 17 17:48:19.290762 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:48:19.290741    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4frmm\" (UniqueName: \"kubernetes.io/projected/eb97e72b-2549-4c22-889e-bacf215d6b26-kube-api-access-4frmm\") pod \"s3-init-djfmf\" (UID: \"eb97e72b-2549-4c22-889e-bacf215d6b26\") " pod="kserve/s3-init-djfmf"
Sep 17 17:48:19.371132 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:48:19.371076    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-djfmf"
Sep 17 17:48:19.659752 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:48:19.659723    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-djfmf"]
Sep 17 17:48:19.662259 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:48:19.662238    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podeb97e72b_2549_4c22_889e_bacf215d6b26.slice/crio-3f04c6f5293d719974ac12a95b39665a8010cdc757bc2a749447c7426a7c4f80 WatchSource:0}: Error finding container 3f04c6f5293d719974ac12a95b39665a8010cdc757bc2a749447c7426a7c4f80: Status 404 returned error can't find the container with id 3f04c6f5293d719974ac12a95b39665a8010cdc757bc2a749447c7426a7c4f80
Sep 17 17:48:20.158871 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:48:20.158833    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-djfmf" event={"ID":"eb97e72b-2549-4c22-889e-bacf215d6b26","Type":"ContainerStarted","Data":"3f04c6f5293d719974ac12a95b39665a8010cdc757bc2a749447c7426a7c4f80"}
Sep 17 17:48:24.180843 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:48:24.180807    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-djfmf" event={"ID":"eb97e72b-2549-4c22-889e-bacf215d6b26","Type":"ContainerStarted","Data":"48d440bf4907207ec3ba4e00e6a89c673993f09a7edd17204ea951bb3de69308"}
Sep 17 17:48:26.112734 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:48:26.112705    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 17:48:26.190838 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:48:26.190807    2567 generic.go:356] "Generic (PLEG): container finished" podID="eb97e72b-2549-4c22-889e-bacf215d6b26" containerID="48d440bf4907207ec3ba4e00e6a89c673993f09a7edd17204ea951bb3de69308" exitCode=0
Sep 17 17:48:26.191007 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:48:26.190851    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-djfmf" event={"ID":"eb97e72b-2549-4c22-889e-bacf215d6b26","Type":"ContainerDied","Data":"48d440bf4907207ec3ba4e00e6a89c673993f09a7edd17204ea951bb3de69308"}
Sep 17 17:48:31.215458 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:48:31.215421    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-djfmf" event={"ID":"eb97e72b-2549-4c22-889e-bacf215d6b26","Type":"ContainerStarted","Data":"e9cb170758316e829206013d042be1247c92391b2365d9de1ba6429dfe88426f"}
Sep 17 17:48:31.237557 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:48:31.237493    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-init-djfmf" podStartSLOduration=1.5690609960000002 podStartE2EDuration="12.23747441s" totalImagesPullingTime="8.03447918s" totalInitContainerRuntime="2.633934234s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:48:19 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:48:31.236384592 +0000 UTC m=+452.259989727" watchObservedRunningTime="2026-09-17 17:48:31.23747441 +0000 UTC m=+452.261079546"
Sep 17 17:48:46.612063 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:48:46.612026    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 17:49:34.346277 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:49:34.346231    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 17:49:34.446849 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:49:34.446816    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 17:49:36.468284 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:49:36.468242    2567 generic.go:356] "Generic (PLEG): container finished" podID="eb97e72b-2549-4c22-889e-bacf215d6b26" containerID="e9cb170758316e829206013d042be1247c92391b2365d9de1ba6429dfe88426f" exitCode=0
Sep 17 17:49:36.468673 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:49:36.468291    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-djfmf" event={"ID":"eb97e72b-2549-4c22-889e-bacf215d6b26","Type":"ContainerDied","Data":"e9cb170758316e829206013d042be1247c92391b2365d9de1ba6429dfe88426f"}
Sep 17 17:49:37.594918 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:49:37.594894    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-djfmf"
Sep 17 17:49:37.608393 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:49:37.608376    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eb97e72b-2549-4c22-889e-bacf215d6b26-model-cache\") pod \"eb97e72b-2549-4c22-889e-bacf215d6b26\" (UID: \"eb97e72b-2549-4c22-889e-bacf215d6b26\") "
Sep 17 17:49:37.608480 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:49:37.608444    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4frmm\" (UniqueName: \"kubernetes.io/projected/eb97e72b-2549-4c22-889e-bacf215d6b26-kube-api-access-4frmm\") pod \"eb97e72b-2549-4c22-889e-bacf215d6b26\" (UID: \"eb97e72b-2549-4c22-889e-bacf215d6b26\") "
Sep 17 17:49:37.610489 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:49:37.610462    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb97e72b-2549-4c22-889e-bacf215d6b26-kube-api-access-4frmm" (OuterVolumeSpecName: "kube-api-access-4frmm") pod "eb97e72b-2549-4c22-889e-bacf215d6b26" (UID: "eb97e72b-2549-4c22-889e-bacf215d6b26"). InnerVolumeSpecName "kube-api-access-4frmm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:49:37.628876 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:49:37.628846    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb97e72b-2549-4c22-889e-bacf215d6b26-model-cache" (OuterVolumeSpecName: "model-cache") pod "eb97e72b-2549-4c22-889e-bacf215d6b26" (UID: "eb97e72b-2549-4c22-889e-bacf215d6b26"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:49:37.709217 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:49:37.709187    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4frmm\" (UniqueName: \"kubernetes.io/projected/eb97e72b-2549-4c22-889e-bacf215d6b26-kube-api-access-4frmm\") on node \"ip-10-0-139-237.ec2.internal\" DevicePath \"\""
Sep 17 17:49:37.709217 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:49:37.709210    2567 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eb97e72b-2549-4c22-889e-bacf215d6b26-model-cache\") on node \"ip-10-0-139-237.ec2.internal\" DevicePath \"\""
Sep 17 17:49:38.477108 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:49:38.477081    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-djfmf"
Sep 17 17:49:38.477108 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:49:38.477092    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-djfmf" event={"ID":"eb97e72b-2549-4c22-889e-bacf215d6b26","Type":"ContainerDied","Data":"3f04c6f5293d719974ac12a95b39665a8010cdc757bc2a749447c7426a7c4f80"}
Sep 17 17:49:38.477299 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:49:38.477124    2567 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3f04c6f5293d719974ac12a95b39665a8010cdc757bc2a749447c7426a7c4f80"
Sep 17 17:49:38.953025 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:49:38.952993    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 17:49:39.046434 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:49:39.046396    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 17:49:50.362729 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:49:50.362644    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 17:49:50.474855 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:49:50.474823    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-74fc5cfbf9-4bx9x"]
Sep 17 17:49:50.475176 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:49:50.475161    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eb97e72b-2549-4c22-889e-bacf215d6b26" containerName="s3-init"
Sep 17 17:49:50.475246 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:49:50.475178    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="eb97e72b-2549-4c22-889e-bacf215d6b26" containerName="s3-init"
Sep 17 17:49:50.475246 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:49:50.475197    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eb97e72b-2549-4c22-889e-bacf215d6b26" containerName="download-hf-model"
Sep 17 17:49:50.475246 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:49:50.475203    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="eb97e72b-2549-4c22-889e-bacf215d6b26" containerName="download-hf-model"
Sep 17 17:49:50.475336 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:49:50.475255    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="eb97e72b-2549-4c22-889e-bacf215d6b26" containerName="s3-init"
Sep 17 17:49:50.478154 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:49:50.478138    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-74fc5cfbf9-4bx9x"
Sep 17 17:49:50.480932 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:49:50.480897    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-1-predictor-serving-cert\""
Sep 17 17:49:50.481055 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:49:50.480932    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-1-kube-rbac-proxy-sar-config\""
Sep 17 17:49:50.481127 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:49:50.481066    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 17:49:50.481185 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:49:50.481136    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-9cm6p\""
Sep 17 17:49:50.481185 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:49:50.481136    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 17:49:50.481993 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:49:50.481970    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 17:49:50.482115 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:49:50.481995    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 17:49:50.489821 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:49:50.489786    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-74fc5cfbf9-4bx9x"]
Sep 17 17:49:50.518939 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:49:50.518912    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m6nvf\" (UniqueName: \"kubernetes.io/projected/e68679b4-02fe-4048-b466-dcb4dd11da7a-kube-api-access-m6nvf\") pod \"isvc-sklearn-graph-1-predictor-74fc5cfbf9-4bx9x\" (UID: \"e68679b4-02fe-4048-b466-dcb4dd11da7a\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-74fc5cfbf9-4bx9x"
Sep 17 17:49:50.519087 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:49:50.518969    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e68679b4-02fe-4048-b466-dcb4dd11da7a-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-1-predictor-74fc5cfbf9-4bx9x\" (UID: \"e68679b4-02fe-4048-b466-dcb4dd11da7a\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-74fc5cfbf9-4bx9x"
Sep 17 17:49:50.519087 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:49:50.519003    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e68679b4-02fe-4048-b466-dcb4dd11da7a-cabundle-cert\") pod \"isvc-sklearn-graph-1-predictor-74fc5cfbf9-4bx9x\" (UID: \"e68679b4-02fe-4048-b466-dcb4dd11da7a\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-74fc5cfbf9-4bx9x"
Sep 17 17:49:50.519087 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:49:50.519034    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e68679b4-02fe-4048-b466-dcb4dd11da7a-proxy-tls\") pod \"isvc-sklearn-graph-1-predictor-74fc5cfbf9-4bx9x\" (UID: \"e68679b4-02fe-4048-b466-dcb4dd11da7a\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-74fc5cfbf9-4bx9x"
Sep 17 17:49:50.519087 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:49:50.519061    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e68679b4-02fe-4048-b466-dcb4dd11da7a-kserve-provision-location\") pod \"isvc-sklearn-graph-1-predictor-74fc5cfbf9-4bx9x\" (UID: \"e68679b4-02fe-4048-b466-dcb4dd11da7a\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-74fc5cfbf9-4bx9x"
Sep 17 17:49:50.567444 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:49:50.567402    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 17:49:50.620396 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:49:50.620319    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e68679b4-02fe-4048-b466-dcb4dd11da7a-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-1-predictor-74fc5cfbf9-4bx9x\" (UID: \"e68679b4-02fe-4048-b466-dcb4dd11da7a\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-74fc5cfbf9-4bx9x"
Sep 17 17:49:50.620627 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:49:50.620605    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e68679b4-02fe-4048-b466-dcb4dd11da7a-cabundle-cert\") pod \"isvc-sklearn-graph-1-predictor-74fc5cfbf9-4bx9x\" (UID: \"e68679b4-02fe-4048-b466-dcb4dd11da7a\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-74fc5cfbf9-4bx9x"
Sep 17 17:49:50.620771 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:49:50.620755    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e68679b4-02fe-4048-b466-dcb4dd11da7a-proxy-tls\") pod \"isvc-sklearn-graph-1-predictor-74fc5cfbf9-4bx9x\" (UID: \"e68679b4-02fe-4048-b466-dcb4dd11da7a\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-74fc5cfbf9-4bx9x"
Sep 17 17:49:50.620894 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:49:50.620880    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e68679b4-02fe-4048-b466-dcb4dd11da7a-kserve-provision-location\") pod \"isvc-sklearn-graph-1-predictor-74fc5cfbf9-4bx9x\" (UID: \"e68679b4-02fe-4048-b466-dcb4dd11da7a\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-74fc5cfbf9-4bx9x"
Sep 17 17:49:50.621050 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:49:50.621036    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m6nvf\" (UniqueName: \"kubernetes.io/projected/e68679b4-02fe-4048-b466-dcb4dd11da7a-kube-api-access-m6nvf\") pod \"isvc-sklearn-graph-1-predictor-74fc5cfbf9-4bx9x\" (UID: \"e68679b4-02fe-4048-b466-dcb4dd11da7a\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-74fc5cfbf9-4bx9x"
Sep 17 17:49:50.621191 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:49:50.621033    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e68679b4-02fe-4048-b466-dcb4dd11da7a-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-1-predictor-74fc5cfbf9-4bx9x\" (UID: \"e68679b4-02fe-4048-b466-dcb4dd11da7a\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-74fc5cfbf9-4bx9x"
Sep 17 17:49:50.625340 ip-10-0-139-237 kubenswrapper[2567]: E0917 17:49:50.621292    2567 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-serving-cert: secret "isvc-sklearn-graph-1-predictor-serving-cert" not found
Sep 17 17:49:50.625340 ip-10-0-139-237 kubenswrapper[2567]: E0917 17:49:50.621388    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e68679b4-02fe-4048-b466-dcb4dd11da7a-proxy-tls podName:e68679b4-02fe-4048-b466-dcb4dd11da7a nodeName:}" failed. No retries permitted until 2026-09-17 17:49:51.121368493 +0000 UTC m=+532.144973621 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/e68679b4-02fe-4048-b466-dcb4dd11da7a-proxy-tls") pod "isvc-sklearn-graph-1-predictor-74fc5cfbf9-4bx9x" (UID: "e68679b4-02fe-4048-b466-dcb4dd11da7a") : secret "isvc-sklearn-graph-1-predictor-serving-cert" not found
Sep 17 17:49:50.625340 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:49:50.621599    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e68679b4-02fe-4048-b466-dcb4dd11da7a-kserve-provision-location\") pod \"isvc-sklearn-graph-1-predictor-74fc5cfbf9-4bx9x\" (UID: \"e68679b4-02fe-4048-b466-dcb4dd11da7a\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-74fc5cfbf9-4bx9x"
Sep 17 17:49:50.625340 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:49:50.621802    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e68679b4-02fe-4048-b466-dcb4dd11da7a-cabundle-cert\") pod \"isvc-sklearn-graph-1-predictor-74fc5cfbf9-4bx9x\" (UID: \"e68679b4-02fe-4048-b466-dcb4dd11da7a\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-74fc5cfbf9-4bx9x"
Sep 17 17:49:50.630262 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:49:50.630240    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m6nvf\" (UniqueName: \"kubernetes.io/projected/e68679b4-02fe-4048-b466-dcb4dd11da7a-kube-api-access-m6nvf\") pod \"isvc-sklearn-graph-1-predictor-74fc5cfbf9-4bx9x\" (UID: \"e68679b4-02fe-4048-b466-dcb4dd11da7a\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-74fc5cfbf9-4bx9x"
Sep 17 17:49:50.702744 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:49:50.702708    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 17:49:50.754750 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:49:50.754725    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-d46ec-predictor-7dbc68c665-9zjhq"]
Sep 17 17:49:50.758177 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:49:50.758162    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-d46ec-predictor-7dbc68c665-9zjhq"
Sep 17 17:49:50.760881 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:49:50.760856    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-d46ec-predictor-serving-cert\""
Sep 17 17:49:50.761148 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:49:50.761132    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-d46ec-kube-rbac-proxy-sar-config\""
Sep 17 17:49:50.768182 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:49:50.768162    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-d46ec-predictor-7dbc68c665-9zjhq"]
Sep 17 17:49:50.797689 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:49:50.797659    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 17:49:50.822755 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:49:50.822724    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6n2zp\" (UniqueName: \"kubernetes.io/projected/b0855a28-11d1-4103-8d6f-8add0ecb15dd-kube-api-access-6n2zp\") pod \"error-404-isvc-d46ec-predictor-7dbc68c665-9zjhq\" (UID: \"b0855a28-11d1-4103-8d6f-8add0ecb15dd\") " pod="kserve-ci-e2e-test/error-404-isvc-d46ec-predictor-7dbc68c665-9zjhq"
Sep 17 17:49:50.822884 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:49:50.822781    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b0855a28-11d1-4103-8d6f-8add0ecb15dd-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-d46ec-predictor-7dbc68c665-9zjhq\" (UID: \"b0855a28-11d1-4103-8d6f-8add0ecb15dd\") " pod="kserve-ci-e2e-test/error-404-isvc-d46ec-predictor-7dbc68c665-9zjhq"
Sep 17 17:49:50.822884 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:49:50.822859    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b0855a28-11d1-4103-8d6f-8add0ecb15dd-proxy-tls\") pod \"error-404-isvc-d46ec-predictor-7dbc68c665-9zjhq\" (UID: \"b0855a28-11d1-4103-8d6f-8add0ecb15dd\") " pod="kserve-ci-e2e-test/error-404-isvc-d46ec-predictor-7dbc68c665-9zjhq"
Sep 17 17:49:50.862195 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:49:50.862162    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 17:49:50.880929 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:49:50.880859    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-xzhr5"]
Sep 17 17:49:50.884271 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:49:50.884254    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-xzhr5"
Sep 17 17:49:50.886813 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:49:50.886793    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-predictor-serving-cert\""
Sep 17 17:49:50.886813 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:49:50.886805    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-kube-rbac-proxy-sar-config\""
Sep 17 17:49:50.899091 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:49:50.899071    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-xzhr5"]
Sep 17 17:49:50.923653 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:49:50.923625    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b0855a28-11d1-4103-8d6f-8add0ecb15dd-proxy-tls\") pod \"error-404-isvc-d46ec-predictor-7dbc68c665-9zjhq\" (UID: \"b0855a28-11d1-4103-8d6f-8add0ecb15dd\") " pod="kserve-ci-e2e-test/error-404-isvc-d46ec-predictor-7dbc68c665-9zjhq"
Sep 17 17:49:50.923754 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:49:50.923684    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/496d8e39-534e-4318-be60-0f872782e0c4-kserve-provision-location\") pod \"isvc-xgboost-graph-predictor-555fc69d56-xzhr5\" (UID: \"496d8e39-534e-4318-be60-0f872782e0c4\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-xzhr5"
Sep 17 17:49:50.923816 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:49:50.923753    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/496d8e39-534e-4318-be60-0f872782e0c4-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-predictor-555fc69d56-xzhr5\" (UID: \"496d8e39-534e-4318-be60-0f872782e0c4\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-xzhr5"
Sep 17 17:49:50.923816 ip-10-0-139-237 kubenswrapper[2567]: E0917 17:49:50.923774    2567 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-d46ec-predictor-serving-cert: secret "error-404-isvc-d46ec-predictor-serving-cert" not found
Sep 17 17:49:50.923816 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:49:50.923790    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/496d8e39-534e-4318-be60-0f872782e0c4-cabundle-cert\") pod \"isvc-xgboost-graph-predictor-555fc69d56-xzhr5\" (UID: \"496d8e39-534e-4318-be60-0f872782e0c4\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-xzhr5"
Sep 17 17:49:50.924066 ip-10-0-139-237 kubenswrapper[2567]: E0917 17:49:50.923851    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b0855a28-11d1-4103-8d6f-8add0ecb15dd-proxy-tls podName:b0855a28-11d1-4103-8d6f-8add0ecb15dd nodeName:}" failed. No retries permitted until 2026-09-17 17:49:51.423829067 +0000 UTC m=+532.447434195 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/b0855a28-11d1-4103-8d6f-8add0ecb15dd-proxy-tls") pod "error-404-isvc-d46ec-predictor-7dbc68c665-9zjhq" (UID: "b0855a28-11d1-4103-8d6f-8add0ecb15dd") : secret "error-404-isvc-d46ec-predictor-serving-cert" not found
Sep 17 17:49:50.924066 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:49:50.923897    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m87s8\" (UniqueName: \"kubernetes.io/projected/496d8e39-534e-4318-be60-0f872782e0c4-kube-api-access-m87s8\") pod \"isvc-xgboost-graph-predictor-555fc69d56-xzhr5\" (UID: \"496d8e39-534e-4318-be60-0f872782e0c4\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-xzhr5"
Sep 17 17:49:50.924066 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:49:50.923960    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6n2zp\" (UniqueName: \"kubernetes.io/projected/b0855a28-11d1-4103-8d6f-8add0ecb15dd-kube-api-access-6n2zp\") pod \"error-404-isvc-d46ec-predictor-7dbc68c665-9zjhq\" (UID: \"b0855a28-11d1-4103-8d6f-8add0ecb15dd\") " pod="kserve-ci-e2e-test/error-404-isvc-d46ec-predictor-7dbc68c665-9zjhq"
Sep 17 17:49:50.924066 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:49:50.924034    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b0855a28-11d1-4103-8d6f-8add0ecb15dd-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-d46ec-predictor-7dbc68c665-9zjhq\" (UID: \"b0855a28-11d1-4103-8d6f-8add0ecb15dd\") " pod="kserve-ci-e2e-test/error-404-isvc-d46ec-predictor-7dbc68c665-9zjhq"
Sep 17 17:49:50.924066 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:49:50.924060    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/496d8e39-534e-4318-be60-0f872782e0c4-proxy-tls\") pod \"isvc-xgboost-graph-predictor-555fc69d56-xzhr5\" (UID: \"496d8e39-534e-4318-be60-0f872782e0c4\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-xzhr5"
Sep 17 17:49:50.924736 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:49:50.924719    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b0855a28-11d1-4103-8d6f-8add0ecb15dd-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-d46ec-predictor-7dbc68c665-9zjhq\" (UID: \"b0855a28-11d1-4103-8d6f-8add0ecb15dd\") " pod="kserve-ci-e2e-test/error-404-isvc-d46ec-predictor-7dbc68c665-9zjhq"
Sep 17 17:49:50.934819 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:49:50.934799    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6n2zp\" (UniqueName: \"kubernetes.io/projected/b0855a28-11d1-4103-8d6f-8add0ecb15dd-kube-api-access-6n2zp\") pod \"error-404-isvc-d46ec-predictor-7dbc68c665-9zjhq\" (UID: \"b0855a28-11d1-4103-8d6f-8add0ecb15dd\") " pod="kserve-ci-e2e-test/error-404-isvc-d46ec-predictor-7dbc68c665-9zjhq"
Sep 17 17:49:51.011605 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:49:51.011557    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 17:49:51.024742 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:49:51.024717    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/496d8e39-534e-4318-be60-0f872782e0c4-kserve-provision-location\") pod \"isvc-xgboost-graph-predictor-555fc69d56-xzhr5\" (UID: \"496d8e39-534e-4318-be60-0f872782e0c4\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-xzhr5"
Sep 17 17:49:51.024891 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:49:51.024755    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/496d8e39-534e-4318-be60-0f872782e0c4-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-predictor-555fc69d56-xzhr5\" (UID: \"496d8e39-534e-4318-be60-0f872782e0c4\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-xzhr5"
Sep 17 17:49:51.024891 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:49:51.024778    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/496d8e39-534e-4318-be60-0f872782e0c4-cabundle-cert\") pod \"isvc-xgboost-graph-predictor-555fc69d56-xzhr5\" (UID: \"496d8e39-534e-4318-be60-0f872782e0c4\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-xzhr5"
Sep 17 17:49:51.024891 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:49:51.024801    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m87s8\" (UniqueName: \"kubernetes.io/projected/496d8e39-534e-4318-be60-0f872782e0c4-kube-api-access-m87s8\") pod \"isvc-xgboost-graph-predictor-555fc69d56-xzhr5\" (UID: \"496d8e39-534e-4318-be60-0f872782e0c4\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-xzhr5"
Sep 17 17:49:51.025054 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:49:51.024892    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/496d8e39-534e-4318-be60-0f872782e0c4-proxy-tls\") pod \"isvc-xgboost-graph-predictor-555fc69d56-xzhr5\" (UID: \"496d8e39-534e-4318-be60-0f872782e0c4\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-xzhr5"
Sep 17 17:49:51.025054 ip-10-0-139-237 kubenswrapper[2567]: E0917 17:49:51.025029    2567 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-xgboost-graph-predictor-serving-cert: secret "isvc-xgboost-graph-predictor-serving-cert" not found
Sep 17 17:49:51.025159 ip-10-0-139-237 kubenswrapper[2567]: E0917 17:49:51.025100    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/496d8e39-534e-4318-be60-0f872782e0c4-proxy-tls podName:496d8e39-534e-4318-be60-0f872782e0c4 nodeName:}" failed. No retries permitted until 2026-09-17 17:49:51.525084747 +0000 UTC m=+532.548689866 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/496d8e39-534e-4318-be60-0f872782e0c4-proxy-tls") pod "isvc-xgboost-graph-predictor-555fc69d56-xzhr5" (UID: "496d8e39-534e-4318-be60-0f872782e0c4") : secret "isvc-xgboost-graph-predictor-serving-cert" not found
Sep 17 17:49:51.025243 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:49:51.025222    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/496d8e39-534e-4318-be60-0f872782e0c4-kserve-provision-location\") pod \"isvc-xgboost-graph-predictor-555fc69d56-xzhr5\" (UID: \"496d8e39-534e-4318-be60-0f872782e0c4\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-xzhr5"
Sep 17 17:49:51.025490 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:49:51.025474    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/496d8e39-534e-4318-be60-0f872782e0c4-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-predictor-555fc69d56-xzhr5\" (UID: \"496d8e39-534e-4318-be60-0f872782e0c4\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-xzhr5"
Sep 17 17:49:51.025543 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:49:51.025474    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/496d8e39-534e-4318-be60-0f872782e0c4-cabundle-cert\") pod \"isvc-xgboost-graph-predictor-555fc69d56-xzhr5\" (UID: \"496d8e39-534e-4318-be60-0f872782e0c4\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-xzhr5"
Sep 17 17:49:51.034002 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:49:51.033978    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m87s8\" (UniqueName: \"kubernetes.io/projected/496d8e39-534e-4318-be60-0f872782e0c4-kube-api-access-m87s8\") pod \"isvc-xgboost-graph-predictor-555fc69d56-xzhr5\" (UID: \"496d8e39-534e-4318-be60-0f872782e0c4\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-xzhr5"
Sep 17 17:49:51.072303 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:49:51.072275    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-76857dd9b4-cvqt8"]
Sep 17 17:49:51.076937 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:49:51.076923    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-76857dd9b4-cvqt8"
Sep 17 17:49:51.079504 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:49:51.079482    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-2-predictor-serving-cert\""
Sep 17 17:49:51.079504 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:49:51.079503    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-2-kube-rbac-proxy-sar-config\""
Sep 17 17:49:51.086101 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:49:51.086077    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-76857dd9b4-cvqt8"]
Sep 17 17:49:51.090347 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:49:51.090327    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 17:49:51.126158 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:49:51.126136    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mx2st\" (UniqueName: \"kubernetes.io/projected/0130260a-d6c1-4b98-81d9-0ee4554b0a5a-kube-api-access-mx2st\") pod \"isvc-sklearn-graph-2-predictor-76857dd9b4-cvqt8\" (UID: \"0130260a-d6c1-4b98-81d9-0ee4554b0a5a\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-76857dd9b4-cvqt8"
Sep 17 17:49:51.126270 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:49:51.126193    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0130260a-d6c1-4b98-81d9-0ee4554b0a5a-kserve-provision-location\") pod \"isvc-sklearn-graph-2-predictor-76857dd9b4-cvqt8\" (UID: \"0130260a-d6c1-4b98-81d9-0ee4554b0a5a\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-76857dd9b4-cvqt8"
Sep 17 17:49:51.126270 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:49:51.126249    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e68679b4-02fe-4048-b466-dcb4dd11da7a-proxy-tls\") pod \"isvc-sklearn-graph-1-predictor-74fc5cfbf9-4bx9x\" (UID: \"e68679b4-02fe-4048-b466-dcb4dd11da7a\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-74fc5cfbf9-4bx9x"
Sep 17 17:49:51.126379 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:49:51.126290    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0130260a-d6c1-4b98-81d9-0ee4554b0a5a-proxy-tls\") pod \"isvc-sklearn-graph-2-predictor-76857dd9b4-cvqt8\" (UID: \"0130260a-d6c1-4b98-81d9-0ee4554b0a5a\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-76857dd9b4-cvqt8"
Sep 17 17:49:51.126379 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:49:51.126375    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0130260a-d6c1-4b98-81d9-0ee4554b0a5a-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-2-predictor-76857dd9b4-cvqt8\" (UID: \"0130260a-d6c1-4b98-81d9-0ee4554b0a5a\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-76857dd9b4-cvqt8"
Sep 17 17:49:51.126479 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:49:51.126405    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0130260a-d6c1-4b98-81d9-0ee4554b0a5a-cabundle-cert\") pod \"isvc-sklearn-graph-2-predictor-76857dd9b4-cvqt8\" (UID: \"0130260a-d6c1-4b98-81d9-0ee4554b0a5a\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-76857dd9b4-cvqt8"
Sep 17 17:49:51.128591 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:49:51.128571    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e68679b4-02fe-4048-b466-dcb4dd11da7a-proxy-tls\") pod \"isvc-sklearn-graph-1-predictor-74fc5cfbf9-4bx9x\" (UID: \"e68679b4-02fe-4048-b466-dcb4dd11da7a\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-74fc5cfbf9-4bx9x"
Sep 17 17:49:51.227228 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:49:51.227151    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0130260a-d6c1-4b98-81d9-0ee4554b0a5a-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-2-predictor-76857dd9b4-cvqt8\" (UID: \"0130260a-d6c1-4b98-81d9-0ee4554b0a5a\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-76857dd9b4-cvqt8"
Sep 17 17:49:51.227228 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:49:51.227193    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0130260a-d6c1-4b98-81d9-0ee4554b0a5a-cabundle-cert\") pod \"isvc-sklearn-graph-2-predictor-76857dd9b4-cvqt8\" (UID: \"0130260a-d6c1-4b98-81d9-0ee4554b0a5a\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-76857dd9b4-cvqt8"
Sep 17 17:49:51.227228 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:49:51.227219    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mx2st\" (UniqueName: \"kubernetes.io/projected/0130260a-d6c1-4b98-81d9-0ee4554b0a5a-kube-api-access-mx2st\") pod \"isvc-sklearn-graph-2-predictor-76857dd9b4-cvqt8\" (UID: \"0130260a-d6c1-4b98-81d9-0ee4554b0a5a\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-76857dd9b4-cvqt8"
Sep 17 17:49:51.227476 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:49:51.227266    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0130260a-d6c1-4b98-81d9-0ee4554b0a5a-kserve-provision-location\") pod \"isvc-sklearn-graph-2-predictor-76857dd9b4-cvqt8\" (UID: \"0130260a-d6c1-4b98-81d9-0ee4554b0a5a\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-76857dd9b4-cvqt8"
Sep 17 17:49:51.227476 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:49:51.227301    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0130260a-d6c1-4b98-81d9-0ee4554b0a5a-proxy-tls\") pod \"isvc-sklearn-graph-2-predictor-76857dd9b4-cvqt8\" (UID: \"0130260a-d6c1-4b98-81d9-0ee4554b0a5a\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-76857dd9b4-cvqt8"
Sep 17 17:49:51.227476 ip-10-0-139-237 kubenswrapper[2567]: E0917 17:49:51.227425    2567 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-serving-cert: secret "isvc-sklearn-graph-2-predictor-serving-cert" not found
Sep 17 17:49:51.227660 ip-10-0-139-237 kubenswrapper[2567]: E0917 17:49:51.227493    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0130260a-d6c1-4b98-81d9-0ee4554b0a5a-proxy-tls podName:0130260a-d6c1-4b98-81d9-0ee4554b0a5a nodeName:}" failed. No retries permitted until 2026-09-17 17:49:51.727474558 +0000 UTC m=+532.751079691 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/0130260a-d6c1-4b98-81d9-0ee4554b0a5a-proxy-tls") pod "isvc-sklearn-graph-2-predictor-76857dd9b4-cvqt8" (UID: "0130260a-d6c1-4b98-81d9-0ee4554b0a5a") : secret "isvc-sklearn-graph-2-predictor-serving-cert" not found
Sep 17 17:49:51.227723 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:49:51.227684    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0130260a-d6c1-4b98-81d9-0ee4554b0a5a-kserve-provision-location\") pod \"isvc-sklearn-graph-2-predictor-76857dd9b4-cvqt8\" (UID: \"0130260a-d6c1-4b98-81d9-0ee4554b0a5a\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-76857dd9b4-cvqt8"
Sep 17 17:49:51.227853 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:49:51.227834    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0130260a-d6c1-4b98-81d9-0ee4554b0a5a-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-2-predictor-76857dd9b4-cvqt8\" (UID: \"0130260a-d6c1-4b98-81d9-0ee4554b0a5a\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-76857dd9b4-cvqt8"
Sep 17 17:49:51.227908 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:49:51.227894    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0130260a-d6c1-4b98-81d9-0ee4554b0a5a-cabundle-cert\") pod \"isvc-sklearn-graph-2-predictor-76857dd9b4-cvqt8\" (UID: \"0130260a-d6c1-4b98-81d9-0ee4554b0a5a\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-76857dd9b4-cvqt8"
Sep 17 17:49:51.235657 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:49:51.235637    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mx2st\" (UniqueName: \"kubernetes.io/projected/0130260a-d6c1-4b98-81d9-0ee4554b0a5a-kube-api-access-mx2st\") pod \"isvc-sklearn-graph-2-predictor-76857dd9b4-cvqt8\" (UID: \"0130260a-d6c1-4b98-81d9-0ee4554b0a5a\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-76857dd9b4-cvqt8"
Sep 17 17:49:51.388838 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:49:51.388804    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-74fc5cfbf9-4bx9x"
Sep 17 17:49:51.403486 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:49:51.403457    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 17:49:51.430027 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:49:51.429985    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b0855a28-11d1-4103-8d6f-8add0ecb15dd-proxy-tls\") pod \"error-404-isvc-d46ec-predictor-7dbc68c665-9zjhq\" (UID: \"b0855a28-11d1-4103-8d6f-8add0ecb15dd\") " pod="kserve-ci-e2e-test/error-404-isvc-d46ec-predictor-7dbc68c665-9zjhq"
Sep 17 17:49:51.432331 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:49:51.432301    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b0855a28-11d1-4103-8d6f-8add0ecb15dd-proxy-tls\") pod \"error-404-isvc-d46ec-predictor-7dbc68c665-9zjhq\" (UID: \"b0855a28-11d1-4103-8d6f-8add0ecb15dd\") " pod="kserve-ci-e2e-test/error-404-isvc-d46ec-predictor-7dbc68c665-9zjhq"
Sep 17 17:49:51.530633 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:49:51.530525    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/496d8e39-534e-4318-be60-0f872782e0c4-proxy-tls\") pod \"isvc-xgboost-graph-predictor-555fc69d56-xzhr5\" (UID: \"496d8e39-534e-4318-be60-0f872782e0c4\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-xzhr5"
Sep 17 17:49:51.533018 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:49:51.532987    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/496d8e39-534e-4318-be60-0f872782e0c4-proxy-tls\") pod \"isvc-xgboost-graph-predictor-555fc69d56-xzhr5\" (UID: \"496d8e39-534e-4318-be60-0f872782e0c4\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-xzhr5"
Sep 17 17:49:51.570717 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:49:51.570687    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 17:49:51.669195 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:49:51.669163    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-d46ec-predictor-7dbc68c665-9zjhq"
Sep 17 17:49:51.679496 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:49:51.679466    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-74fc5cfbf9-4bx9x"]
Sep 17 17:49:51.681565 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:49:51.681520    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode68679b4_02fe_4048_b466_dcb4dd11da7a.slice/crio-a3a1bfe5a4849bf7348edba349b80e582a253749aefba85d606e7453a5f44036 WatchSource:0}: Error finding container a3a1bfe5a4849bf7348edba349b80e582a253749aefba85d606e7453a5f44036: Status 404 returned error can't find the container with id a3a1bfe5a4849bf7348edba349b80e582a253749aefba85d606e7453a5f44036
Sep 17 17:49:51.733276 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:49:51.733250    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0130260a-d6c1-4b98-81d9-0ee4554b0a5a-proxy-tls\") pod \"isvc-sklearn-graph-2-predictor-76857dd9b4-cvqt8\" (UID: \"0130260a-d6c1-4b98-81d9-0ee4554b0a5a\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-76857dd9b4-cvqt8"
Sep 17 17:49:51.735733 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:49:51.735709    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0130260a-d6c1-4b98-81d9-0ee4554b0a5a-proxy-tls\") pod \"isvc-sklearn-graph-2-predictor-76857dd9b4-cvqt8\" (UID: \"0130260a-d6c1-4b98-81d9-0ee4554b0a5a\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-76857dd9b4-cvqt8"
Sep 17 17:49:51.794272 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:49:51.794248    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-xzhr5"
Sep 17 17:49:51.946022 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:49:51.945992    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-d46ec-predictor-7dbc68c665-9zjhq"]
Sep 17 17:49:51.949152 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:49:51.949121    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0855a28_11d1_4103_8d6f_8add0ecb15dd.slice/crio-c9d068706305432c4ba5c09d6c129548d4f7356195762fa5adf437a12070950c WatchSource:0}: Error finding container c9d068706305432c4ba5c09d6c129548d4f7356195762fa5adf437a12070950c: Status 404 returned error can't find the container with id c9d068706305432c4ba5c09d6c129548d4f7356195762fa5adf437a12070950c
Sep 17 17:49:51.987832 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:49:51.987806    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-76857dd9b4-cvqt8"
Sep 17 17:49:52.083917 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:49:52.083845    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-xzhr5"]
Sep 17 17:49:52.087965 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:49:52.087937    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod496d8e39_534e_4318_be60_0f872782e0c4.slice/crio-319feca9d1c5972c4536094de427485d6d8b7256e73b5f0636fd5020ba4279fe WatchSource:0}: Error finding container 319feca9d1c5972c4536094de427485d6d8b7256e73b5f0636fd5020ba4279fe: Status 404 returned error can't find the container with id 319feca9d1c5972c4536094de427485d6d8b7256e73b5f0636fd5020ba4279fe
Sep 17 17:49:52.263239 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:49:52.263212    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-76857dd9b4-cvqt8"]
Sep 17 17:49:52.266407 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:49:52.266376    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0130260a_d6c1_4b98_81d9_0ee4554b0a5a.slice/crio-ecf83a1446f5237671e46d0eb0ec29bb50a84fae95e7f6be534914847bfdccf3 WatchSource:0}: Error finding container ecf83a1446f5237671e46d0eb0ec29bb50a84fae95e7f6be534914847bfdccf3: Status 404 returned error can't find the container with id ecf83a1446f5237671e46d0eb0ec29bb50a84fae95e7f6be534914847bfdccf3
Sep 17 17:49:52.535888 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:49:52.535165    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-xzhr5" event={"ID":"496d8e39-534e-4318-be60-0f872782e0c4","Type":"ContainerStarted","Data":"1f212862496e96cdee944ccdeca1dd541b81a9d1a0897014f3d155696f0a97c2"}
Sep 17 17:49:52.535888 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:49:52.535240    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-xzhr5" event={"ID":"496d8e39-534e-4318-be60-0f872782e0c4","Type":"ContainerStarted","Data":"319feca9d1c5972c4536094de427485d6d8b7256e73b5f0636fd5020ba4279fe"}
Sep 17 17:49:52.538961 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:49:52.538930    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-74fc5cfbf9-4bx9x" event={"ID":"e68679b4-02fe-4048-b466-dcb4dd11da7a","Type":"ContainerStarted","Data":"7de5f07df5ddf975d8972815785d02c0382e3db33db2bdbf3bdc9fe24a06c4cc"}
Sep 17 17:49:52.539123 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:49:52.538995    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-74fc5cfbf9-4bx9x" event={"ID":"e68679b4-02fe-4048-b466-dcb4dd11da7a","Type":"ContainerStarted","Data":"a3a1bfe5a4849bf7348edba349b80e582a253749aefba85d606e7453a5f44036"}
Sep 17 17:49:52.544670 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:49:52.543022    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-76857dd9b4-cvqt8" event={"ID":"0130260a-d6c1-4b98-81d9-0ee4554b0a5a","Type":"ContainerStarted","Data":"251c6264618654049ff9979487e9296c6168fbd5a0c3db6ffbc5354424098227"}
Sep 17 17:49:52.544670 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:49:52.543084    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-76857dd9b4-cvqt8" event={"ID":"0130260a-d6c1-4b98-81d9-0ee4554b0a5a","Type":"ContainerStarted","Data":"ecf83a1446f5237671e46d0eb0ec29bb50a84fae95e7f6be534914847bfdccf3"}
Sep 17 17:49:52.546861 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:49:52.546793    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-d46ec-predictor-7dbc68c665-9zjhq" event={"ID":"b0855a28-11d1-4103-8d6f-8add0ecb15dd","Type":"ContainerStarted","Data":"c9d068706305432c4ba5c09d6c129548d4f7356195762fa5adf437a12070950c"}
Sep 17 17:49:56.573642 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:49:56.573554    2567 generic.go:356] "Generic (PLEG): container finished" podID="e68679b4-02fe-4048-b466-dcb4dd11da7a" containerID="7de5f07df5ddf975d8972815785d02c0382e3db33db2bdbf3bdc9fe24a06c4cc" exitCode=0
Sep 17 17:49:56.573642 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:49:56.573616    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-74fc5cfbf9-4bx9x" event={"ID":"e68679b4-02fe-4048-b466-dcb4dd11da7a","Type":"ContainerDied","Data":"7de5f07df5ddf975d8972815785d02c0382e3db33db2bdbf3bdc9fe24a06c4cc"}
Sep 17 17:49:57.584640 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:49:57.583567    2567 generic.go:356] "Generic (PLEG): container finished" podID="496d8e39-534e-4318-be60-0f872782e0c4" containerID="1f212862496e96cdee944ccdeca1dd541b81a9d1a0897014f3d155696f0a97c2" exitCode=0
Sep 17 17:49:57.584640 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:49:57.583686    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-xzhr5" event={"ID":"496d8e39-534e-4318-be60-0f872782e0c4","Type":"ContainerDied","Data":"1f212862496e96cdee944ccdeca1dd541b81a9d1a0897014f3d155696f0a97c2"}
Sep 17 17:49:57.590121 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:49:57.590091    2567 generic.go:356] "Generic (PLEG): container finished" podID="0130260a-d6c1-4b98-81d9-0ee4554b0a5a" containerID="251c6264618654049ff9979487e9296c6168fbd5a0c3db6ffbc5354424098227" exitCode=0
Sep 17 17:49:57.590234 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:49:57.590167    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-76857dd9b4-cvqt8" event={"ID":"0130260a-d6c1-4b98-81d9-0ee4554b0a5a","Type":"ContainerDied","Data":"251c6264618654049ff9979487e9296c6168fbd5a0c3db6ffbc5354424098227"}
Sep 17 17:50:09.662003 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:50:09.661963    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-76857dd9b4-cvqt8" event={"ID":"0130260a-d6c1-4b98-81d9-0ee4554b0a5a","Type":"ContainerStarted","Data":"d1fd737280f68bc5993dde3b7257c4733dba2faf0eb8005641aeabe79f0fcf05"}
Sep 17 17:50:09.663768 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:50:09.663736    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-d46ec-predictor-7dbc68c665-9zjhq" event={"ID":"b0855a28-11d1-4103-8d6f-8add0ecb15dd","Type":"ContainerStarted","Data":"2c8f654fac26c43e6cdd75ecfa17bba945959f2162d23fc4f4b4db24b5161233"}
Sep 17 17:50:09.665842 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:50:09.665810    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-74fc5cfbf9-4bx9x" event={"ID":"e68679b4-02fe-4048-b466-dcb4dd11da7a","Type":"ContainerStarted","Data":"e6d053e1149269fa0a2d9f93ac9e1b177701254add208af0bf2c90a730a679ee"}
Sep 17 17:50:14.703895 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:50:14.703845    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-d46ec-predictor-7dbc68c665-9zjhq" event={"ID":"b0855a28-11d1-4103-8d6f-8add0ecb15dd","Type":"ContainerStarted","Data":"54f1b5154f3f87da4068d8efcb6dc9fe9248db980d3f04c6f1a78692edb5f4ed"}
Sep 17 17:50:14.704503 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:50:14.704048    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-d46ec-predictor-7dbc68c665-9zjhq"
Sep 17 17:50:14.706938 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:50:14.706901    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-74fc5cfbf9-4bx9x" event={"ID":"e68679b4-02fe-4048-b466-dcb4dd11da7a","Type":"ContainerStarted","Data":"02fad77b74cd3a78ee4791c8b7d0cb32bcdd4b429711cb1cf139ad7dc46ed57b"}
Sep 17 17:50:14.707065 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:50:14.706985    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-74fc5cfbf9-4bx9x"
Sep 17 17:50:14.707065 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:50:14.707033    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-74fc5cfbf9-4bx9x"
Sep 17 17:50:14.708798 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:50:14.708756    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-74fc5cfbf9-4bx9x" podUID="e68679b4-02fe-4048-b466-dcb4dd11da7a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Sep 17 17:50:14.709681 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:50:14.709658    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-76857dd9b4-cvqt8" event={"ID":"0130260a-d6c1-4b98-81d9-0ee4554b0a5a","Type":"ContainerStarted","Data":"2d9d5fbe3dcdc35839cf4baafc69a129ba230c85142ff895333f0bb4fbefff2d"}
Sep 17 17:50:14.709834 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:50:14.709806    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-76857dd9b4-cvqt8"
Sep 17 17:50:14.726908 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:50:14.726856    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-d46ec-predictor-7dbc68c665-9zjhq" podStartSLOduration=2.857564163 podStartE2EDuration="24.726842165s" totalImagesPullingTime="21.869278002s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:49:50 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:50:14.724145511 +0000 UTC m=+555.747750660" watchObservedRunningTime="2026-09-17 17:50:14.726842165 +0000 UTC m=+555.750447300"
Sep 17 17:50:14.746072 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:50:14.746032    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-76857dd9b4-cvqt8" podStartSLOduration=3.30117498 podStartE2EDuration="23.746018692s" totalImagesPullingTime="16.276175592s" totalInitContainerRuntime="4.16866812s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:49:51 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:50:14.744147545 +0000 UTC m=+555.767752711" watchObservedRunningTime="2026-09-17 17:50:14.746018692 +0000 UTC m=+555.769623827"
Sep 17 17:50:14.768668 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:50:14.768625    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-74fc5cfbf9-4bx9x" podStartSLOduration=3.797967593 podStartE2EDuration="24.768608023s" totalImagesPullingTime="17.230146897s" totalInitContainerRuntime="3.740493533s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:49:50 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:50:14.766256012 +0000 UTC m=+555.789861170" watchObservedRunningTime="2026-09-17 17:50:14.768608023 +0000 UTC m=+555.792213159"
Sep 17 17:50:15.715239 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:50:15.715186    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-74fc5cfbf9-4bx9x" podUID="e68679b4-02fe-4048-b466-dcb4dd11da7a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Sep 17 17:50:15.715699 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:50:15.715209    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-d46ec-predictor-7dbc68c665-9zjhq"
Sep 17 17:50:15.715699 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:50:15.715319    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-76857dd9b4-cvqt8"
Sep 17 17:50:15.716071 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:50:15.715977    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-d46ec-predictor-7dbc68c665-9zjhq" podUID="b0855a28-11d1-4103-8d6f-8add0ecb15dd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Sep 17 17:50:15.716183 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:50:15.716146    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-76857dd9b4-cvqt8" podUID="0130260a-d6c1-4b98-81d9-0ee4554b0a5a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Sep 17 17:50:16.719689 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:50:16.719645    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-76857dd9b4-cvqt8" podUID="0130260a-d6c1-4b98-81d9-0ee4554b0a5a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Sep 17 17:50:16.720110 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:50:16.719645    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-d46ec-predictor-7dbc68c665-9zjhq" podUID="b0855a28-11d1-4103-8d6f-8add0ecb15dd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Sep 17 17:50:20.721384 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:50:20.721352    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-74fc5cfbf9-4bx9x"
Sep 17 17:50:20.722053 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:50:20.722023    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-74fc5cfbf9-4bx9x" podUID="e68679b4-02fe-4048-b466-dcb4dd11da7a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Sep 17 17:50:21.724036 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:50:21.724009    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-76857dd9b4-cvqt8"
Sep 17 17:50:21.724451 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:50:21.724101    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-d46ec-predictor-7dbc68c665-9zjhq"
Sep 17 17:50:21.724582 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:50:21.724556    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-d46ec-predictor-7dbc68c665-9zjhq" podUID="b0855a28-11d1-4103-8d6f-8add0ecb15dd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Sep 17 17:50:21.724845 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:50:21.724818    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-76857dd9b4-cvqt8" podUID="0130260a-d6c1-4b98-81d9-0ee4554b0a5a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Sep 17 17:50:30.722564 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:50:30.722507    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-74fc5cfbf9-4bx9x" podUID="e68679b4-02fe-4048-b466-dcb4dd11da7a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Sep 17 17:50:31.724760 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:50:31.724726    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-d46ec-predictor-7dbc68c665-9zjhq" podUID="b0855a28-11d1-4103-8d6f-8add0ecb15dd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Sep 17 17:50:31.725074 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:50:31.724815    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-76857dd9b4-cvqt8" podUID="0130260a-d6c1-4b98-81d9-0ee4554b0a5a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Sep 17 17:50:31.793593 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:50:31.793564    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-xzhr5" event={"ID":"496d8e39-534e-4318-be60-0f872782e0c4","Type":"ContainerStarted","Data":"ab30fee80574d6269309bf06605b2ecb0de302fbeb6fe08a62a36443ec1dbece"}
Sep 17 17:50:32.803281 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:50:32.803240    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-xzhr5" event={"ID":"496d8e39-534e-4318-be60-0f872782e0c4","Type":"ContainerStarted","Data":"7327999b4426e99567e59a7d1f82ca7093aa838e94de41a7b2d9cd45dfc85810"}
Sep 17 17:50:32.803860 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:50:32.803840    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-xzhr5"
Sep 17 17:50:32.803921 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:50:32.803895    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-xzhr5"
Sep 17 17:50:32.805339 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:50:32.805314    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-xzhr5" podUID="496d8e39-534e-4318-be60-0f872782e0c4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Sep 17 17:50:32.827692 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:50:32.827654    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-xzhr5" podStartSLOduration=4.663345013 podStartE2EDuration="42.827641929s" totalImagesPullingTime="34.002358446s" totalInitContainerRuntime="4.16193847s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:49:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:50:32.826732003 +0000 UTC m=+573.850337138" watchObservedRunningTime="2026-09-17 17:50:32.827641929 +0000 UTC m=+573.851247064"
Sep 17 17:50:33.807747 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:50:33.807698    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-xzhr5" podUID="496d8e39-534e-4318-be60-0f872782e0c4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Sep 17 17:50:38.812802 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:50:38.812771    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-xzhr5"
Sep 17 17:50:38.813438 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:50:38.813408    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-xzhr5" podUID="496d8e39-534e-4318-be60-0f872782e0c4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Sep 17 17:50:40.722219 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:50:40.722177    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-74fc5cfbf9-4bx9x" podUID="e68679b4-02fe-4048-b466-dcb4dd11da7a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Sep 17 17:50:41.725275 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:50:41.725239    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-d46ec-predictor-7dbc68c665-9zjhq" podUID="b0855a28-11d1-4103-8d6f-8add0ecb15dd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Sep 17 17:50:41.725661 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:50:41.725239    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-76857dd9b4-cvqt8" podUID="0130260a-d6c1-4b98-81d9-0ee4554b0a5a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Sep 17 17:50:48.814157 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:50:48.814119    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-xzhr5" podUID="496d8e39-534e-4318-be60-0f872782e0c4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Sep 17 17:50:50.722905 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:50:50.722865    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-74fc5cfbf9-4bx9x" podUID="e68679b4-02fe-4048-b466-dcb4dd11da7a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Sep 17 17:50:51.725045 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:50:51.725012    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-d46ec-predictor-7dbc68c665-9zjhq" podUID="b0855a28-11d1-4103-8d6f-8add0ecb15dd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Sep 17 17:50:51.725420 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:50:51.725015    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-76857dd9b4-cvqt8" podUID="0130260a-d6c1-4b98-81d9-0ee4554b0a5a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Sep 17 17:50:58.814228 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:50:58.814191    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-xzhr5" podUID="496d8e39-534e-4318-be60-0f872782e0c4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Sep 17 17:51:00.722701 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:51:00.722661    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-74fc5cfbf9-4bx9x" podUID="e68679b4-02fe-4048-b466-dcb4dd11da7a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Sep 17 17:51:01.724898 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:51:01.724855    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-76857dd9b4-cvqt8" podUID="0130260a-d6c1-4b98-81d9-0ee4554b0a5a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Sep 17 17:51:01.725448 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:51:01.725429    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-d46ec-predictor-7dbc68c665-9zjhq"
Sep 17 17:51:03.354425 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:51:03.354382    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 17:51:08.813907 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:51:08.813860    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-xzhr5" podUID="496d8e39-534e-4318-be60-0f872782e0c4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Sep 17 17:51:10.362899 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:51:10.362857    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 17:51:10.722347 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:51:10.722258    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-74fc5cfbf9-4bx9x" podUID="e68679b4-02fe-4048-b466-dcb4dd11da7a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Sep 17 17:51:10.765961 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:51:10.765930    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 17:51:11.724819 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:51:11.724778    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-76857dd9b4-cvqt8" podUID="0130260a-d6c1-4b98-81d9-0ee4554b0a5a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Sep 17 17:51:18.813807 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:51:18.813767    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-xzhr5" podUID="496d8e39-534e-4318-be60-0f872782e0c4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Sep 17 17:51:20.579153 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:51:20.579122    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 17:51:20.671261 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:51:20.671227    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-d46ec-predictor-7dbc68c665-9zjhq"]
Sep 17 17:51:20.671513 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:51:20.671488    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-d46ec-predictor-7dbc68c665-9zjhq" podUID="b0855a28-11d1-4103-8d6f-8add0ecb15dd" containerName="kserve-container" containerID="cri-o://2c8f654fac26c43e6cdd75ecfa17bba945959f2162d23fc4f4b4db24b5161233" gracePeriod=30
Sep 17 17:51:20.671635 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:51:20.671549    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-d46ec-predictor-7dbc68c665-9zjhq" podUID="b0855a28-11d1-4103-8d6f-8add0ecb15dd" containerName="kube-rbac-proxy" containerID="cri-o://54f1b5154f3f87da4068d8efcb6dc9fe9248db980d3f04c6f1a78692edb5f4ed" gracePeriod=30
Sep 17 17:51:20.722617 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:51:20.722575    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-74fc5cfbf9-4bx9x" podUID="e68679b4-02fe-4048-b466-dcb4dd11da7a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Sep 17 17:51:20.879362 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:51:20.879288    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-8592d-predictor-94446b57c-nnlbx"]
Sep 17 17:51:20.881628 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:51:20.881613    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-8592d-predictor-94446b57c-nnlbx"
Sep 17 17:51:20.884468 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:51:20.884445    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-8592d-predictor-serving-cert\""
Sep 17 17:51:20.884596 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:51:20.884452    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-8592d-kube-rbac-proxy-sar-config\""
Sep 17 17:51:20.893766 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:51:20.893743    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-8592d-predictor-94446b57c-nnlbx"]
Sep 17 17:51:20.928390 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:51:20.928365    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fb8979a8-6a3e-47e6-9c2e-9f59a659a84e-proxy-tls\") pod \"error-404-isvc-8592d-predictor-94446b57c-nnlbx\" (UID: \"fb8979a8-6a3e-47e6-9c2e-9f59a659a84e\") " pod="kserve-ci-e2e-test/error-404-isvc-8592d-predictor-94446b57c-nnlbx"
Sep 17 17:51:20.928487 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:51:20.928405    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fb8979a8-6a3e-47e6-9c2e-9f59a659a84e-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-8592d-predictor-94446b57c-nnlbx\" (UID: \"fb8979a8-6a3e-47e6-9c2e-9f59a659a84e\") " pod="kserve-ci-e2e-test/error-404-isvc-8592d-predictor-94446b57c-nnlbx"
Sep 17 17:51:20.928487 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:51:20.928479    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8k6q6\" (UniqueName: \"kubernetes.io/projected/fb8979a8-6a3e-47e6-9c2e-9f59a659a84e-kube-api-access-8k6q6\") pod \"error-404-isvc-8592d-predictor-94446b57c-nnlbx\" (UID: \"fb8979a8-6a3e-47e6-9c2e-9f59a659a84e\") " pod="kserve-ci-e2e-test/error-404-isvc-8592d-predictor-94446b57c-nnlbx"
Sep 17 17:51:21.009614 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:51:21.009583    2567 generic.go:356] "Generic (PLEG): container finished" podID="b0855a28-11d1-4103-8d6f-8add0ecb15dd" containerID="54f1b5154f3f87da4068d8efcb6dc9fe9248db980d3f04c6f1a78692edb5f4ed" exitCode=2
Sep 17 17:51:21.009773 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:51:21.009650    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-d46ec-predictor-7dbc68c665-9zjhq" event={"ID":"b0855a28-11d1-4103-8d6f-8add0ecb15dd","Type":"ContainerDied","Data":"54f1b5154f3f87da4068d8efcb6dc9fe9248db980d3f04c6f1a78692edb5f4ed"}
Sep 17 17:51:21.028893 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:51:21.028871    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8k6q6\" (UniqueName: \"kubernetes.io/projected/fb8979a8-6a3e-47e6-9c2e-9f59a659a84e-kube-api-access-8k6q6\") pod \"error-404-isvc-8592d-predictor-94446b57c-nnlbx\" (UID: \"fb8979a8-6a3e-47e6-9c2e-9f59a659a84e\") " pod="kserve-ci-e2e-test/error-404-isvc-8592d-predictor-94446b57c-nnlbx"
Sep 17 17:51:21.028981 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:51:21.028914    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fb8979a8-6a3e-47e6-9c2e-9f59a659a84e-proxy-tls\") pod \"error-404-isvc-8592d-predictor-94446b57c-nnlbx\" (UID: \"fb8979a8-6a3e-47e6-9c2e-9f59a659a84e\") " pod="kserve-ci-e2e-test/error-404-isvc-8592d-predictor-94446b57c-nnlbx"
Sep 17 17:51:21.028981 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:51:21.028961    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fb8979a8-6a3e-47e6-9c2e-9f59a659a84e-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-8592d-predictor-94446b57c-nnlbx\" (UID: \"fb8979a8-6a3e-47e6-9c2e-9f59a659a84e\") " pod="kserve-ci-e2e-test/error-404-isvc-8592d-predictor-94446b57c-nnlbx"
Sep 17 17:51:21.029675 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:51:21.029654    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fb8979a8-6a3e-47e6-9c2e-9f59a659a84e-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-8592d-predictor-94446b57c-nnlbx\" (UID: \"fb8979a8-6a3e-47e6-9c2e-9f59a659a84e\") " pod="kserve-ci-e2e-test/error-404-isvc-8592d-predictor-94446b57c-nnlbx"
Sep 17 17:51:21.031265 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:51:21.031243    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fb8979a8-6a3e-47e6-9c2e-9f59a659a84e-proxy-tls\") pod \"error-404-isvc-8592d-predictor-94446b57c-nnlbx\" (UID: \"fb8979a8-6a3e-47e6-9c2e-9f59a659a84e\") " pod="kserve-ci-e2e-test/error-404-isvc-8592d-predictor-94446b57c-nnlbx"
Sep 17 17:51:21.037947 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:51:21.037928    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8k6q6\" (UniqueName: \"kubernetes.io/projected/fb8979a8-6a3e-47e6-9c2e-9f59a659a84e-kube-api-access-8k6q6\") pod \"error-404-isvc-8592d-predictor-94446b57c-nnlbx\" (UID: \"fb8979a8-6a3e-47e6-9c2e-9f59a659a84e\") " pod="kserve-ci-e2e-test/error-404-isvc-8592d-predictor-94446b57c-nnlbx"
Sep 17 17:51:21.191816 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:51:21.191733    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-8592d-predictor-94446b57c-nnlbx"
Sep 17 17:51:21.466632 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:51:21.466610    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-8592d-predictor-94446b57c-nnlbx"]
Sep 17 17:51:21.468857 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:51:21.468827    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb8979a8_6a3e_47e6_9c2e_9f59a659a84e.slice/crio-d886469da838904c9e6f65002390cc3f08570381efcf9df0370df5db56361d80 WatchSource:0}: Error finding container d886469da838904c9e6f65002390cc3f08570381efcf9df0370df5db56361d80: Status 404 returned error can't find the container with id d886469da838904c9e6f65002390cc3f08570381efcf9df0370df5db56361d80
Sep 17 17:51:21.720044 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:51:21.719946    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-d46ec-predictor-7dbc68c665-9zjhq" podUID="b0855a28-11d1-4103-8d6f-8add0ecb15dd" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.26:8643/healthz\": dial tcp 10.134.0.26:8643: connect: connection refused"
Sep 17 17:51:21.724692 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:51:21.724659    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-d46ec-predictor-7dbc68c665-9zjhq" podUID="b0855a28-11d1-4103-8d6f-8add0ecb15dd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Sep 17 17:51:21.724821 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:51:21.724749    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-76857dd9b4-cvqt8" podUID="0130260a-d6c1-4b98-81d9-0ee4554b0a5a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Sep 17 17:51:22.016291 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:51:22.016202    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-8592d-predictor-94446b57c-nnlbx" event={"ID":"fb8979a8-6a3e-47e6-9c2e-9f59a659a84e","Type":"ContainerStarted","Data":"1285cb2af78097681eda517bf1a3bb962e0c27967ad434e579f8975ae301458d"}
Sep 17 17:51:22.016291 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:51:22.016284    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-8592d-predictor-94446b57c-nnlbx"
Sep 17 17:51:22.016521 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:51:22.016314    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-8592d-predictor-94446b57c-nnlbx"
Sep 17 17:51:22.016521 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:51:22.016334    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-8592d-predictor-94446b57c-nnlbx" event={"ID":"fb8979a8-6a3e-47e6-9c2e-9f59a659a84e","Type":"ContainerStarted","Data":"3fab16144dd478b86aeb537ef3a198d1440239226d9445ad5ffa7e8c7d13e416"}
Sep 17 17:51:22.016521 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:51:22.016352    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-8592d-predictor-94446b57c-nnlbx" event={"ID":"fb8979a8-6a3e-47e6-9c2e-9f59a659a84e","Type":"ContainerStarted","Data":"d886469da838904c9e6f65002390cc3f08570381efcf9df0370df5db56361d80"}
Sep 17 17:51:22.017686 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:51:22.017661    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-8592d-predictor-94446b57c-nnlbx" podUID="fb8979a8-6a3e-47e6-9c2e-9f59a659a84e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Sep 17 17:51:22.281869 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:51:22.281775    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-76857dd9b4-cvqt8" podUID="0130260a-d6c1-4b98-81d9-0ee4554b0a5a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Sep 17 17:51:22.868030 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:51:22.867998    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 17:51:23.020452 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:51:23.020417    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-8592d-predictor-94446b57c-nnlbx" podUID="fb8979a8-6a3e-47e6-9c2e-9f59a659a84e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Sep 17 17:51:24.409983 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:51:24.409958    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-d46ec-predictor-7dbc68c665-9zjhq"
Sep 17 17:51:24.430607 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:51:24.430567    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-8592d-predictor-94446b57c-nnlbx" podStartSLOduration=4.430548378 podStartE2EDuration="4.430548378s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:51:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:51:22.059663729 +0000 UTC m=+623.083268891" watchObservedRunningTime="2026-09-17 17:51:24.430548378 +0000 UTC m=+625.454153515"
Sep 17 17:51:24.460521 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:51:24.460485    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6n2zp\" (UniqueName: \"kubernetes.io/projected/b0855a28-11d1-4103-8d6f-8add0ecb15dd-kube-api-access-6n2zp\") pod \"b0855a28-11d1-4103-8d6f-8add0ecb15dd\" (UID: \"b0855a28-11d1-4103-8d6f-8add0ecb15dd\") "
Sep 17 17:51:24.460678 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:51:24.460623    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b0855a28-11d1-4103-8d6f-8add0ecb15dd-proxy-tls\") pod \"b0855a28-11d1-4103-8d6f-8add0ecb15dd\" (UID: \"b0855a28-11d1-4103-8d6f-8add0ecb15dd\") "
Sep 17 17:51:24.460678 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:51:24.460674    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b0855a28-11d1-4103-8d6f-8add0ecb15dd-kube-rbac-proxy-sar-config\") pod \"b0855a28-11d1-4103-8d6f-8add0ecb15dd\" (UID: \"b0855a28-11d1-4103-8d6f-8add0ecb15dd\") "
Sep 17 17:51:24.461092 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:51:24.461060    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0855a28-11d1-4103-8d6f-8add0ecb15dd-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "b0855a28-11d1-4103-8d6f-8add0ecb15dd" (UID: "b0855a28-11d1-4103-8d6f-8add0ecb15dd"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:51:24.462710 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:51:24.462685    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b0855a28-11d1-4103-8d6f-8add0ecb15dd-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "b0855a28-11d1-4103-8d6f-8add0ecb15dd" (UID: "b0855a28-11d1-4103-8d6f-8add0ecb15dd"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:51:24.462808 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:51:24.462688    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0855a28-11d1-4103-8d6f-8add0ecb15dd-kube-api-access-6n2zp" (OuterVolumeSpecName: "kube-api-access-6n2zp") pod "b0855a28-11d1-4103-8d6f-8add0ecb15dd" (UID: "b0855a28-11d1-4103-8d6f-8add0ecb15dd"). InnerVolumeSpecName "kube-api-access-6n2zp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:51:24.561464 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:51:24.561431    2567 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b0855a28-11d1-4103-8d6f-8add0ecb15dd-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-237.ec2.internal\" DevicePath \"\""
Sep 17 17:51:24.561464 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:51:24.561462    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6n2zp\" (UniqueName: \"kubernetes.io/projected/b0855a28-11d1-4103-8d6f-8add0ecb15dd-kube-api-access-6n2zp\") on node \"ip-10-0-139-237.ec2.internal\" DevicePath \"\""
Sep 17 17:51:24.561464 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:51:24.561472    2567 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b0855a28-11d1-4103-8d6f-8add0ecb15dd-proxy-tls\") on node \"ip-10-0-139-237.ec2.internal\" DevicePath \"\""
Sep 17 17:51:25.030431 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:51:25.030343    2567 generic.go:356] "Generic (PLEG): container finished" podID="b0855a28-11d1-4103-8d6f-8add0ecb15dd" containerID="2c8f654fac26c43e6cdd75ecfa17bba945959f2162d23fc4f4b4db24b5161233" exitCode=0
Sep 17 17:51:25.030431 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:51:25.030390    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-d46ec-predictor-7dbc68c665-9zjhq"
Sep 17 17:51:25.030658 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:51:25.030426    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-d46ec-predictor-7dbc68c665-9zjhq" event={"ID":"b0855a28-11d1-4103-8d6f-8add0ecb15dd","Type":"ContainerDied","Data":"2c8f654fac26c43e6cdd75ecfa17bba945959f2162d23fc4f4b4db24b5161233"}
Sep 17 17:51:25.030658 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:51:25.030484    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-d46ec-predictor-7dbc68c665-9zjhq" event={"ID":"b0855a28-11d1-4103-8d6f-8add0ecb15dd","Type":"ContainerDied","Data":"c9d068706305432c4ba5c09d6c129548d4f7356195762fa5adf437a12070950c"}
Sep 17 17:51:25.030658 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:51:25.030511    2567 scope.go:117] "RemoveContainer" containerID="54f1b5154f3f87da4068d8efcb6dc9fe9248db980d3f04c6f1a78692edb5f4ed"
Sep 17 17:51:25.040342 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:51:25.040329    2567 scope.go:117] "RemoveContainer" containerID="2c8f654fac26c43e6cdd75ecfa17bba945959f2162d23fc4f4b4db24b5161233"
Sep 17 17:51:25.047444 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:51:25.047428    2567 scope.go:117] "RemoveContainer" containerID="54f1b5154f3f87da4068d8efcb6dc9fe9248db980d3f04c6f1a78692edb5f4ed"
Sep 17 17:51:25.047703 ip-10-0-139-237 kubenswrapper[2567]: E0917 17:51:25.047684    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"54f1b5154f3f87da4068d8efcb6dc9fe9248db980d3f04c6f1a78692edb5f4ed\": container with ID starting with 54f1b5154f3f87da4068d8efcb6dc9fe9248db980d3f04c6f1a78692edb5f4ed not found: ID does not exist" containerID="54f1b5154f3f87da4068d8efcb6dc9fe9248db980d3f04c6f1a78692edb5f4ed"
Sep 17 17:51:25.047757 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:51:25.047711    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54f1b5154f3f87da4068d8efcb6dc9fe9248db980d3f04c6f1a78692edb5f4ed"} err="failed to get container status \"54f1b5154f3f87da4068d8efcb6dc9fe9248db980d3f04c6f1a78692edb5f4ed\": rpc error: code = NotFound desc = could not find container \"54f1b5154f3f87da4068d8efcb6dc9fe9248db980d3f04c6f1a78692edb5f4ed\": container with ID starting with 54f1b5154f3f87da4068d8efcb6dc9fe9248db980d3f04c6f1a78692edb5f4ed not found: ID does not exist"
Sep 17 17:51:25.047757 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:51:25.047726    2567 scope.go:117] "RemoveContainer" containerID="2c8f654fac26c43e6cdd75ecfa17bba945959f2162d23fc4f4b4db24b5161233"
Sep 17 17:51:25.047949 ip-10-0-139-237 kubenswrapper[2567]: E0917 17:51:25.047932    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2c8f654fac26c43e6cdd75ecfa17bba945959f2162d23fc4f4b4db24b5161233\": container with ID starting with 2c8f654fac26c43e6cdd75ecfa17bba945959f2162d23fc4f4b4db24b5161233 not found: ID does not exist" containerID="2c8f654fac26c43e6cdd75ecfa17bba945959f2162d23fc4f4b4db24b5161233"
Sep 17 17:51:25.047999 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:51:25.047959    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c8f654fac26c43e6cdd75ecfa17bba945959f2162d23fc4f4b4db24b5161233"} err="failed to get container status \"2c8f654fac26c43e6cdd75ecfa17bba945959f2162d23fc4f4b4db24b5161233\": rpc error: code = NotFound desc = could not find container \"2c8f654fac26c43e6cdd75ecfa17bba945959f2162d23fc4f4b4db24b5161233\": container with ID starting with 2c8f654fac26c43e6cdd75ecfa17bba945959f2162d23fc4f4b4db24b5161233 not found: ID does not exist"
Sep 17 17:51:25.054438 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:51:25.054408    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-d46ec-predictor-7dbc68c665-9zjhq"]
Sep 17 17:51:25.058562 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:51:25.057250    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-d46ec-predictor-7dbc68c665-9zjhq"]
Sep 17 17:51:25.292867 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:51:25.292836    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b0855a28-11d1-4103-8d6f-8add0ecb15dd" path="/var/lib/kubelet/pods/b0855a28-11d1-4103-8d6f-8add0ecb15dd/volumes"
Sep 17 17:51:28.026005 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:51:28.025763    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-8592d-predictor-94446b57c-nnlbx"
Sep 17 17:51:28.026434 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:51:28.026276    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-8592d-predictor-94446b57c-nnlbx" podUID="fb8979a8-6a3e-47e6-9c2e-9f59a659a84e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Sep 17 17:51:28.813592 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:51:28.813554    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-xzhr5" podUID="496d8e39-534e-4318-be60-0f872782e0c4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Sep 17 17:51:30.722596 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:51:30.722559    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-74fc5cfbf9-4bx9x" podUID="e68679b4-02fe-4048-b466-dcb4dd11da7a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Sep 17 17:51:32.282766 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:51:32.282736    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-76857dd9b4-cvqt8"
Sep 17 17:51:37.286842 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:51:37.286812    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-74fc5cfbf9-4bx9x"
Sep 17 17:51:38.026700 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:51:38.026657    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-8592d-predictor-94446b57c-nnlbx" podUID="fb8979a8-6a3e-47e6-9c2e-9f59a659a84e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Sep 17 17:51:38.814660 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:51:38.814627    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-xzhr5"
Sep 17 17:51:48.027190 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:51:48.027152    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-8592d-predictor-94446b57c-nnlbx" podUID="fb8979a8-6a3e-47e6-9c2e-9f59a659a84e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Sep 17 17:51:58.026633 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:51:58.026593    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-8592d-predictor-94446b57c-nnlbx" podUID="fb8979a8-6a3e-47e6-9c2e-9f59a659a84e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Sep 17 17:52:00.375858 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:52:00.375797    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 17:52:00.818890 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:52:00.818862    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 17:52:08.026679 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:52:08.026652    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-8592d-predictor-94446b57c-nnlbx"
Sep 17 17:52:10.959623 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:52:10.959590    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 17:52:11.072371 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:52:11.072336    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-74fc5cfbf9-4bx9x"]
Sep 17 17:52:11.072628 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:52:11.072600    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-74fc5cfbf9-4bx9x" podUID="e68679b4-02fe-4048-b466-dcb4dd11da7a" containerName="kserve-container" containerID="cri-o://e6d053e1149269fa0a2d9f93ac9e1b177701254add208af0bf2c90a730a679ee" gracePeriod=30
Sep 17 17:52:11.072735 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:52:11.072650    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-74fc5cfbf9-4bx9x" podUID="e68679b4-02fe-4048-b466-dcb4dd11da7a" containerName="kube-rbac-proxy" containerID="cri-o://02fad77b74cd3a78ee4791c8b7d0cb32bcdd4b429711cb1cf139ad7dc46ed57b" gracePeriod=30
Sep 17 17:52:11.088386 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:52:11.088359    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 17:52:11.100131 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:52:11.100105    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-xzhr5"]
Sep 17 17:52:11.100376 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:52:11.100319    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-xzhr5" podUID="496d8e39-534e-4318-be60-0f872782e0c4" containerName="kserve-container" containerID="cri-o://ab30fee80574d6269309bf06605b2ecb0de302fbeb6fe08a62a36443ec1dbece" gracePeriod=30
Sep 17 17:52:11.100463 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:52:11.100359    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-xzhr5" podUID="496d8e39-534e-4318-be60-0f872782e0c4" containerName="kube-rbac-proxy" containerID="cri-o://7327999b4426e99567e59a7d1f82ca7093aa838e94de41a7b2d9cd45dfc85810" gracePeriod=30
Sep 17 17:52:11.135282 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:52:11.135256    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-76857dd9b4-cvqt8"]
Sep 17 17:52:11.135493 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:52:11.135468    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-76857dd9b4-cvqt8" podUID="0130260a-d6c1-4b98-81d9-0ee4554b0a5a" containerName="kserve-container" containerID="cri-o://d1fd737280f68bc5993dde3b7257c4733dba2faf0eb8005641aeabe79f0fcf05" gracePeriod=30
Sep 17 17:52:11.135592 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:52:11.135513    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-76857dd9b4-cvqt8" podUID="0130260a-d6c1-4b98-81d9-0ee4554b0a5a" containerName="kube-rbac-proxy" containerID="cri-o://2d9d5fbe3dcdc35839cf4baafc69a129ba230c85142ff895333f0bb4fbefff2d" gracePeriod=30
Sep 17 17:52:11.194988 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:52:11.194961    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 17:52:11.226867 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:52:11.226801    2567 generic.go:356] "Generic (PLEG): container finished" podID="496d8e39-534e-4318-be60-0f872782e0c4" containerID="7327999b4426e99567e59a7d1f82ca7093aa838e94de41a7b2d9cd45dfc85810" exitCode=2
Sep 17 17:52:11.226999 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:52:11.226865    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-xzhr5" event={"ID":"496d8e39-534e-4318-be60-0f872782e0c4","Type":"ContainerDied","Data":"7327999b4426e99567e59a7d1f82ca7093aa838e94de41a7b2d9cd45dfc85810"}
Sep 17 17:52:11.228709 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:52:11.228682    2567 generic.go:356] "Generic (PLEG): container finished" podID="e68679b4-02fe-4048-b466-dcb4dd11da7a" containerID="02fad77b74cd3a78ee4791c8b7d0cb32bcdd4b429711cb1cf139ad7dc46ed57b" exitCode=2
Sep 17 17:52:11.228805 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:52:11.228755    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-74fc5cfbf9-4bx9x" event={"ID":"e68679b4-02fe-4048-b466-dcb4dd11da7a","Type":"ContainerDied","Data":"02fad77b74cd3a78ee4791c8b7d0cb32bcdd4b429711cb1cf139ad7dc46ed57b"}
Sep 17 17:52:11.289332 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:52:11.289289    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 17:52:11.345329 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:52:11.345298    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-081d8-predictor-7f479c548c-h2qk4"]
Sep 17 17:52:11.345665 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:52:11.345652    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b0855a28-11d1-4103-8d6f-8add0ecb15dd" containerName="kserve-container"
Sep 17 17:52:11.345665 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:52:11.345666    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="b0855a28-11d1-4103-8d6f-8add0ecb15dd" containerName="kserve-container"
Sep 17 17:52:11.345766 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:52:11.345687    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b0855a28-11d1-4103-8d6f-8add0ecb15dd" containerName="kube-rbac-proxy"
Sep 17 17:52:11.345766 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:52:11.345692    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="b0855a28-11d1-4103-8d6f-8add0ecb15dd" containerName="kube-rbac-proxy"
Sep 17 17:52:11.345766 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:52:11.345751    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="b0855a28-11d1-4103-8d6f-8add0ecb15dd" containerName="kube-rbac-proxy"
Sep 17 17:52:11.345766 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:52:11.345761    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="b0855a28-11d1-4103-8d6f-8add0ecb15dd" containerName="kserve-container"
Sep 17 17:52:11.348769 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:52:11.348754    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-081d8-predictor-7f479c548c-h2qk4"
Sep 17 17:52:11.351448 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:52:11.351419    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-081d8-predictor-serving-cert\""
Sep 17 17:52:11.351816 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:52:11.351791    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-081d8-kube-rbac-proxy-sar-config\""
Sep 17 17:52:11.361455 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:52:11.361429    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-081d8-predictor-7f479c548c-h2qk4"]
Sep 17 17:52:11.385365 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:52:11.385341    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 17:52:11.450044 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:52:11.450023    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 17:52:11.473205 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:52:11.473165    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b92d1167-af4a-4333-bcfe-73b07e791faf-proxy-tls\") pod \"error-404-isvc-081d8-predictor-7f479c548c-h2qk4\" (UID: \"b92d1167-af4a-4333-bcfe-73b07e791faf\") " pod="kserve-ci-e2e-test/error-404-isvc-081d8-predictor-7f479c548c-h2qk4"
Sep 17 17:52:11.473205 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:52:11.473213    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b92d1167-af4a-4333-bcfe-73b07e791faf-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-081d8-predictor-7f479c548c-h2qk4\" (UID: \"b92d1167-af4a-4333-bcfe-73b07e791faf\") " pod="kserve-ci-e2e-test/error-404-isvc-081d8-predictor-7f479c548c-h2qk4"
Sep 17 17:52:11.473377 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:52:11.473231    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7l9k2\" (UniqueName: \"kubernetes.io/projected/b92d1167-af4a-4333-bcfe-73b07e791faf-kube-api-access-7l9k2\") pod \"error-404-isvc-081d8-predictor-7f479c548c-h2qk4\" (UID: \"b92d1167-af4a-4333-bcfe-73b07e791faf\") " pod="kserve-ci-e2e-test/error-404-isvc-081d8-predictor-7f479c548c-h2qk4"
Sep 17 17:52:11.574729 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:52:11.574688    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b92d1167-af4a-4333-bcfe-73b07e791faf-proxy-tls\") pod \"error-404-isvc-081d8-predictor-7f479c548c-h2qk4\" (UID: \"b92d1167-af4a-4333-bcfe-73b07e791faf\") " pod="kserve-ci-e2e-test/error-404-isvc-081d8-predictor-7f479c548c-h2qk4"
Sep 17 17:52:11.574729 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:52:11.574741    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b92d1167-af4a-4333-bcfe-73b07e791faf-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-081d8-predictor-7f479c548c-h2qk4\" (UID: \"b92d1167-af4a-4333-bcfe-73b07e791faf\") " pod="kserve-ci-e2e-test/error-404-isvc-081d8-predictor-7f479c548c-h2qk4"
Sep 17 17:52:11.574958 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:52:11.574762    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7l9k2\" (UniqueName: \"kubernetes.io/projected/b92d1167-af4a-4333-bcfe-73b07e791faf-kube-api-access-7l9k2\") pod \"error-404-isvc-081d8-predictor-7f479c548c-h2qk4\" (UID: \"b92d1167-af4a-4333-bcfe-73b07e791faf\") " pod="kserve-ci-e2e-test/error-404-isvc-081d8-predictor-7f479c548c-h2qk4"
Sep 17 17:52:11.574958 ip-10-0-139-237 kubenswrapper[2567]: E0917 17:52:11.574848    2567 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-081d8-predictor-serving-cert: secret "error-404-isvc-081d8-predictor-serving-cert" not found
Sep 17 17:52:11.574958 ip-10-0-139-237 kubenswrapper[2567]: E0917 17:52:11.574922    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b92d1167-af4a-4333-bcfe-73b07e791faf-proxy-tls podName:b92d1167-af4a-4333-bcfe-73b07e791faf nodeName:}" failed. No retries permitted until 2026-09-17 17:52:12.074902416 +0000 UTC m=+673.098507533 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/b92d1167-af4a-4333-bcfe-73b07e791faf-proxy-tls") pod "error-404-isvc-081d8-predictor-7f479c548c-h2qk4" (UID: "b92d1167-af4a-4333-bcfe-73b07e791faf") : secret "error-404-isvc-081d8-predictor-serving-cert" not found
Sep 17 17:52:11.575383 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:52:11.575366    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b92d1167-af4a-4333-bcfe-73b07e791faf-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-081d8-predictor-7f479c548c-h2qk4\" (UID: \"b92d1167-af4a-4333-bcfe-73b07e791faf\") " pod="kserve-ci-e2e-test/error-404-isvc-081d8-predictor-7f479c548c-h2qk4"
Sep 17 17:52:11.584372 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:52:11.584342    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7l9k2\" (UniqueName: \"kubernetes.io/projected/b92d1167-af4a-4333-bcfe-73b07e791faf-kube-api-access-7l9k2\") pod \"error-404-isvc-081d8-predictor-7f479c548c-h2qk4\" (UID: \"b92d1167-af4a-4333-bcfe-73b07e791faf\") " pod="kserve-ci-e2e-test/error-404-isvc-081d8-predictor-7f479c548c-h2qk4"
Sep 17 17:52:11.720071 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:52:11.720035    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-76857dd9b4-cvqt8" podUID="0130260a-d6c1-4b98-81d9-0ee4554b0a5a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.28:8643/healthz\": dial tcp 10.134.0.28:8643: connect: connection refused"
Sep 17 17:52:11.882481 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:52:11.882390    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 17:52:12.080419 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:52:12.080382    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b92d1167-af4a-4333-bcfe-73b07e791faf-proxy-tls\") pod \"error-404-isvc-081d8-predictor-7f479c548c-h2qk4\" (UID: \"b92d1167-af4a-4333-bcfe-73b07e791faf\") " pod="kserve-ci-e2e-test/error-404-isvc-081d8-predictor-7f479c548c-h2qk4"
Sep 17 17:52:12.082848 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:52:12.082824    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b92d1167-af4a-4333-bcfe-73b07e791faf-proxy-tls\") pod \"error-404-isvc-081d8-predictor-7f479c548c-h2qk4\" (UID: \"b92d1167-af4a-4333-bcfe-73b07e791faf\") " pod="kserve-ci-e2e-test/error-404-isvc-081d8-predictor-7f479c548c-h2qk4"
Sep 17 17:52:12.235404 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:52:12.235330    2567 generic.go:356] "Generic (PLEG): container finished" podID="0130260a-d6c1-4b98-81d9-0ee4554b0a5a" containerID="2d9d5fbe3dcdc35839cf4baafc69a129ba230c85142ff895333f0bb4fbefff2d" exitCode=2
Sep 17 17:52:12.235556 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:52:12.235401    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-76857dd9b4-cvqt8" event={"ID":"0130260a-d6c1-4b98-81d9-0ee4554b0a5a","Type":"ContainerDied","Data":"2d9d5fbe3dcdc35839cf4baafc69a129ba230c85142ff895333f0bb4fbefff2d"}
Sep 17 17:52:12.259609 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:52:12.259581    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-081d8-predictor-7f479c548c-h2qk4"
Sep 17 17:52:12.281910 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:52:12.281880    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-76857dd9b4-cvqt8" podUID="0130260a-d6c1-4b98-81d9-0ee4554b0a5a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Sep 17 17:52:12.541348 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:52:12.541323    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-081d8-predictor-7f479c548c-h2qk4"]
Sep 17 17:52:12.543308 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:52:12.543283    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb92d1167_af4a_4333_bcfe_73b07e791faf.slice/crio-ad162807f05a57307b60c7a24144b148b957b022b14e527a4427e761a1e9399b WatchSource:0}: Error finding container ad162807f05a57307b60c7a24144b148b957b022b14e527a4427e761a1e9399b: Status 404 returned error can't find the container with id ad162807f05a57307b60c7a24144b148b957b022b14e527a4427e761a1e9399b
Sep 17 17:52:13.241899 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:52:13.241863    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-081d8-predictor-7f479c548c-h2qk4" event={"ID":"b92d1167-af4a-4333-bcfe-73b07e791faf","Type":"ContainerStarted","Data":"cbaa9fb3c40255bf1b4486dde27a4db45aceb4a46951120e7f2de876d025c5b6"}
Sep 17 17:52:13.242366 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:52:13.241930    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-081d8-predictor-7f479c548c-h2qk4" event={"ID":"b92d1167-af4a-4333-bcfe-73b07e791faf","Type":"ContainerStarted","Data":"0dbb7b93cb833b6517a2fdae35aee1a473d69e9a3f58094bd6e409359fc9d8f9"}
Sep 17 17:52:13.242366 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:52:13.241970    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-081d8-predictor-7f479c548c-h2qk4"
Sep 17 17:52:13.242366 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:52:13.242004    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-081d8-predictor-7f479c548c-h2qk4" event={"ID":"b92d1167-af4a-4333-bcfe-73b07e791faf","Type":"ContainerStarted","Data":"ad162807f05a57307b60c7a24144b148b957b022b14e527a4427e761a1e9399b"}
Sep 17 17:52:13.269577 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:52:13.269506    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-081d8-predictor-7f479c548c-h2qk4" podStartSLOduration=2.2694903699999998 podStartE2EDuration="2.26949037s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:52:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:52:13.267624896 +0000 UTC m=+674.291230057" watchObservedRunningTime="2026-09-17 17:52:13.26949037 +0000 UTC m=+674.293095506"
Sep 17 17:52:13.808183 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:52:13.808140    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-xzhr5" podUID="496d8e39-534e-4318-be60-0f872782e0c4" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.27:8643/healthz\": dial tcp 10.134.0.27:8643: connect: connection refused"
Sep 17 17:52:14.245720 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:52:14.245627    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-081d8-predictor-7f479c548c-h2qk4"
Sep 17 17:52:14.246984 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:52:14.246954    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-081d8-predictor-7f479c548c-h2qk4" podUID="b92d1167-af4a-4333-bcfe-73b07e791faf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Sep 17 17:52:15.249362 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:52:15.249322    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-081d8-predictor-7f479c548c-h2qk4" podUID="b92d1167-af4a-4333-bcfe-73b07e791faf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Sep 17 17:52:15.715485 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:52:15.715442    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-74fc5cfbf9-4bx9x" podUID="e68679b4-02fe-4048-b466-dcb4dd11da7a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.25:8643/healthz\": dial tcp 10.134.0.25:8643: connect: connection refused"
Sep 17 17:52:16.041209 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:52:16.041183    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-76857dd9b4-cvqt8"
Sep 17 17:52:16.044183 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:52:16.044160    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-74fc5cfbf9-4bx9x"
Sep 17 17:52:16.115200 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:52:16.115175    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0130260a-d6c1-4b98-81d9-0ee4554b0a5a-proxy-tls\") pod \"0130260a-d6c1-4b98-81d9-0ee4554b0a5a\" (UID: \"0130260a-d6c1-4b98-81d9-0ee4554b0a5a\") "
Sep 17 17:52:16.115368 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:52:16.115224    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0130260a-d6c1-4b98-81d9-0ee4554b0a5a-kube-rbac-proxy-sar-config\") pod \"0130260a-d6c1-4b98-81d9-0ee4554b0a5a\" (UID: \"0130260a-d6c1-4b98-81d9-0ee4554b0a5a\") "
Sep 17 17:52:16.115368 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:52:16.115252    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e68679b4-02fe-4048-b466-dcb4dd11da7a-kserve-provision-location\") pod \"e68679b4-02fe-4048-b466-dcb4dd11da7a\" (UID: \"e68679b4-02fe-4048-b466-dcb4dd11da7a\") "
Sep 17 17:52:16.115368 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:52:16.115320    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e68679b4-02fe-4048-b466-dcb4dd11da7a-cabundle-cert\") pod \"e68679b4-02fe-4048-b466-dcb4dd11da7a\" (UID: \"e68679b4-02fe-4048-b466-dcb4dd11da7a\") "
Sep 17 17:52:16.115562 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:52:16.115383    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m6nvf\" (UniqueName: \"kubernetes.io/projected/e68679b4-02fe-4048-b466-dcb4dd11da7a-kube-api-access-m6nvf\") pod \"e68679b4-02fe-4048-b466-dcb4dd11da7a\" (UID: \"e68679b4-02fe-4048-b466-dcb4dd11da7a\") "
Sep 17 17:52:16.115562 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:52:16.115424    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e68679b4-02fe-4048-b466-dcb4dd11da7a-proxy-tls\") pod \"e68679b4-02fe-4048-b466-dcb4dd11da7a\" (UID: \"e68679b4-02fe-4048-b466-dcb4dd11da7a\") "
Sep 17 17:52:16.115562 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:52:16.115458    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0130260a-d6c1-4b98-81d9-0ee4554b0a5a-kserve-provision-location\") pod \"0130260a-d6c1-4b98-81d9-0ee4554b0a5a\" (UID: \"0130260a-d6c1-4b98-81d9-0ee4554b0a5a\") "
Sep 17 17:52:16.115562 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:52:16.115488    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mx2st\" (UniqueName: \"kubernetes.io/projected/0130260a-d6c1-4b98-81d9-0ee4554b0a5a-kube-api-access-mx2st\") pod \"0130260a-d6c1-4b98-81d9-0ee4554b0a5a\" (UID: \"0130260a-d6c1-4b98-81d9-0ee4554b0a5a\") "
Sep 17 17:52:16.115772 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:52:16.115574    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0130260a-d6c1-4b98-81d9-0ee4554b0a5a-cabundle-cert\") pod \"0130260a-d6c1-4b98-81d9-0ee4554b0a5a\" (UID: \"0130260a-d6c1-4b98-81d9-0ee4554b0a5a\") "
Sep 17 17:52:16.115772 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:52:16.115607    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e68679b4-02fe-4048-b466-dcb4dd11da7a-kube-rbac-proxy-sar-config\") pod \"e68679b4-02fe-4048-b466-dcb4dd11da7a\" (UID: \"e68679b4-02fe-4048-b466-dcb4dd11da7a\") "
Sep 17 17:52:16.115772 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:52:16.115609    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0130260a-d6c1-4b98-81d9-0ee4554b0a5a-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "0130260a-d6c1-4b98-81d9-0ee4554b0a5a" (UID: "0130260a-d6c1-4b98-81d9-0ee4554b0a5a"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:52:16.115772 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:52:16.115675    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e68679b4-02fe-4048-b466-dcb4dd11da7a-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "e68679b4-02fe-4048-b466-dcb4dd11da7a" (UID: "e68679b4-02fe-4048-b466-dcb4dd11da7a"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:52:16.115965 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:52:16.115861    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0130260a-d6c1-4b98-81d9-0ee4554b0a5a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "0130260a-d6c1-4b98-81d9-0ee4554b0a5a" (UID: "0130260a-d6c1-4b98-81d9-0ee4554b0a5a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:52:16.115965 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:52:16.115877    2567 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0130260a-d6c1-4b98-81d9-0ee4554b0a5a-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-237.ec2.internal\" DevicePath \"\""
Sep 17 17:52:16.115965 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:52:16.115895    2567 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e68679b4-02fe-4048-b466-dcb4dd11da7a-cabundle-cert\") on node \"ip-10-0-139-237.ec2.internal\" DevicePath \"\""
Sep 17 17:52:16.115965 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:52:16.115915    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e68679b4-02fe-4048-b466-dcb4dd11da7a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e68679b4-02fe-4048-b466-dcb4dd11da7a" (UID: "e68679b4-02fe-4048-b466-dcb4dd11da7a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:52:16.116168 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:52:16.116133    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0130260a-d6c1-4b98-81d9-0ee4554b0a5a-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "0130260a-d6c1-4b98-81d9-0ee4554b0a5a" (UID: "0130260a-d6c1-4b98-81d9-0ee4554b0a5a"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:52:16.116221 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:52:16.116195    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e68679b4-02fe-4048-b466-dcb4dd11da7a-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "e68679b4-02fe-4048-b466-dcb4dd11da7a" (UID: "e68679b4-02fe-4048-b466-dcb4dd11da7a"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:52:16.117808 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:52:16.117746    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e68679b4-02fe-4048-b466-dcb4dd11da7a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "e68679b4-02fe-4048-b466-dcb4dd11da7a" (UID: "e68679b4-02fe-4048-b466-dcb4dd11da7a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:52:16.117940 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:52:16.117841    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0130260a-d6c1-4b98-81d9-0ee4554b0a5a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "0130260a-d6c1-4b98-81d9-0ee4554b0a5a" (UID: "0130260a-d6c1-4b98-81d9-0ee4554b0a5a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:52:16.118040 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:52:16.118008    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0130260a-d6c1-4b98-81d9-0ee4554b0a5a-kube-api-access-mx2st" (OuterVolumeSpecName: "kube-api-access-mx2st") pod "0130260a-d6c1-4b98-81d9-0ee4554b0a5a" (UID: "0130260a-d6c1-4b98-81d9-0ee4554b0a5a"). InnerVolumeSpecName "kube-api-access-mx2st". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:52:16.118040 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:52:16.118021    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e68679b4-02fe-4048-b466-dcb4dd11da7a-kube-api-access-m6nvf" (OuterVolumeSpecName: "kube-api-access-m6nvf") pod "e68679b4-02fe-4048-b466-dcb4dd11da7a" (UID: "e68679b4-02fe-4048-b466-dcb4dd11da7a"). InnerVolumeSpecName "kube-api-access-m6nvf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:52:16.134269 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:52:16.134247    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-xzhr5"
Sep 17 17:52:16.216203 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:52:16.216167    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m87s8\" (UniqueName: \"kubernetes.io/projected/496d8e39-534e-4318-be60-0f872782e0c4-kube-api-access-m87s8\") pod \"496d8e39-534e-4318-be60-0f872782e0c4\" (UID: \"496d8e39-534e-4318-be60-0f872782e0c4\") "
Sep 17 17:52:16.216405 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:52:16.216223    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/496d8e39-534e-4318-be60-0f872782e0c4-kserve-provision-location\") pod \"496d8e39-534e-4318-be60-0f872782e0c4\" (UID: \"496d8e39-534e-4318-be60-0f872782e0c4\") "
Sep 17 17:52:16.216405 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:52:16.216303    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/496d8e39-534e-4318-be60-0f872782e0c4-cabundle-cert\") pod \"496d8e39-534e-4318-be60-0f872782e0c4\" (UID: \"496d8e39-534e-4318-be60-0f872782e0c4\") "
Sep 17 17:52:16.216405 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:52:16.216351    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/496d8e39-534e-4318-be60-0f872782e0c4-kube-rbac-proxy-sar-config\") pod \"496d8e39-534e-4318-be60-0f872782e0c4\" (UID: \"496d8e39-534e-4318-be60-0f872782e0c4\") "
Sep 17 17:52:16.216669 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:52:16.216420    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/496d8e39-534e-4318-be60-0f872782e0c4-proxy-tls\") pod \"496d8e39-534e-4318-be60-0f872782e0c4\" (UID: \"496d8e39-534e-4318-be60-0f872782e0c4\") "
Sep 17 17:52:16.216669 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:52:16.216546    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/496d8e39-534e-4318-be60-0f872782e0c4-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "496d8e39-534e-4318-be60-0f872782e0c4" (UID: "496d8e39-534e-4318-be60-0f872782e0c4"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:52:16.216769 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:52:16.216665    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/496d8e39-534e-4318-be60-0f872782e0c4-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "496d8e39-534e-4318-be60-0f872782e0c4" (UID: "496d8e39-534e-4318-be60-0f872782e0c4"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:52:16.216769 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:52:16.216737    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/496d8e39-534e-4318-be60-0f872782e0c4-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "496d8e39-534e-4318-be60-0f872782e0c4" (UID: "496d8e39-534e-4318-be60-0f872782e0c4"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:52:16.216907 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:52:16.216817    2567 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e68679b4-02fe-4048-b466-dcb4dd11da7a-kserve-provision-location\") on node \"ip-10-0-139-237.ec2.internal\" DevicePath \"\""
Sep 17 17:52:16.216907 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:52:16.216830    2567 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/496d8e39-534e-4318-be60-0f872782e0c4-kserve-provision-location\") on node \"ip-10-0-139-237.ec2.internal\" DevicePath \"\""
Sep 17 17:52:16.216907 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:52:16.216839    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m6nvf\" (UniqueName: \"kubernetes.io/projected/e68679b4-02fe-4048-b466-dcb4dd11da7a-kube-api-access-m6nvf\") on node \"ip-10-0-139-237.ec2.internal\" DevicePath \"\""
Sep 17 17:52:16.216907 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:52:16.216850    2567 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/496d8e39-534e-4318-be60-0f872782e0c4-cabundle-cert\") on node \"ip-10-0-139-237.ec2.internal\" DevicePath \"\""
Sep 17 17:52:16.216907 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:52:16.216863    2567 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e68679b4-02fe-4048-b466-dcb4dd11da7a-proxy-tls\") on node \"ip-10-0-139-237.ec2.internal\" DevicePath \"\""
Sep 17 17:52:16.216907 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:52:16.216870    2567 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/496d8e39-534e-4318-be60-0f872782e0c4-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-237.ec2.internal\" DevicePath \"\""
Sep 17 17:52:16.216907 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:52:16.216879    2567 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0130260a-d6c1-4b98-81d9-0ee4554b0a5a-kserve-provision-location\") on node \"ip-10-0-139-237.ec2.internal\" DevicePath \"\""
Sep 17 17:52:16.216907 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:52:16.216888    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mx2st\" (UniqueName: \"kubernetes.io/projected/0130260a-d6c1-4b98-81d9-0ee4554b0a5a-kube-api-access-mx2st\") on node \"ip-10-0-139-237.ec2.internal\" DevicePath \"\""
Sep 17 17:52:16.216907 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:52:16.216897    2567 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0130260a-d6c1-4b98-81d9-0ee4554b0a5a-cabundle-cert\") on node \"ip-10-0-139-237.ec2.internal\" DevicePath \"\""
Sep 17 17:52:16.216907 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:52:16.216907    2567 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e68679b4-02fe-4048-b466-dcb4dd11da7a-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-237.ec2.internal\" DevicePath \"\""
Sep 17 17:52:16.217411 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:52:16.216920    2567 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0130260a-d6c1-4b98-81d9-0ee4554b0a5a-proxy-tls\") on node \"ip-10-0-139-237.ec2.internal\" DevicePath \"\""
Sep 17 17:52:16.218464 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:52:16.218435    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/496d8e39-534e-4318-be60-0f872782e0c4-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "496d8e39-534e-4318-be60-0f872782e0c4" (UID: "496d8e39-534e-4318-be60-0f872782e0c4"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:52:16.218593 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:52:16.218510    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/496d8e39-534e-4318-be60-0f872782e0c4-kube-api-access-m87s8" (OuterVolumeSpecName: "kube-api-access-m87s8") pod "496d8e39-534e-4318-be60-0f872782e0c4" (UID: "496d8e39-534e-4318-be60-0f872782e0c4"). InnerVolumeSpecName "kube-api-access-m87s8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:52:16.257020 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:52:16.256984    2567 generic.go:356] "Generic (PLEG): container finished" podID="496d8e39-534e-4318-be60-0f872782e0c4" containerID="ab30fee80574d6269309bf06605b2ecb0de302fbeb6fe08a62a36443ec1dbece" exitCode=0
Sep 17 17:52:16.257399 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:52:16.257030    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-xzhr5" event={"ID":"496d8e39-534e-4318-be60-0f872782e0c4","Type":"ContainerDied","Data":"ab30fee80574d6269309bf06605b2ecb0de302fbeb6fe08a62a36443ec1dbece"}
Sep 17 17:52:16.257399 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:52:16.257061    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-xzhr5"
Sep 17 17:52:16.257399 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:52:16.257131    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-xzhr5" event={"ID":"496d8e39-534e-4318-be60-0f872782e0c4","Type":"ContainerDied","Data":"319feca9d1c5972c4536094de427485d6d8b7256e73b5f0636fd5020ba4279fe"}
Sep 17 17:52:16.257399 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:52:16.257198    2567 scope.go:117] "RemoveContainer" containerID="7327999b4426e99567e59a7d1f82ca7093aa838e94de41a7b2d9cd45dfc85810"
Sep 17 17:52:16.259716 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:52:16.259689    2567 generic.go:356] "Generic (PLEG): container finished" podID="e68679b4-02fe-4048-b466-dcb4dd11da7a" containerID="e6d053e1149269fa0a2d9f93ac9e1b177701254add208af0bf2c90a730a679ee" exitCode=0
Sep 17 17:52:16.259835 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:52:16.259770    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-74fc5cfbf9-4bx9x" event={"ID":"e68679b4-02fe-4048-b466-dcb4dd11da7a","Type":"ContainerDied","Data":"e6d053e1149269fa0a2d9f93ac9e1b177701254add208af0bf2c90a730a679ee"}
Sep 17 17:52:16.259835 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:52:16.259808    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-74fc5cfbf9-4bx9x"
Sep 17 17:52:16.259835 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:52:16.259829    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-74fc5cfbf9-4bx9x" event={"ID":"e68679b4-02fe-4048-b466-dcb4dd11da7a","Type":"ContainerDied","Data":"a3a1bfe5a4849bf7348edba349b80e582a253749aefba85d606e7453a5f44036"}
Sep 17 17:52:16.262390 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:52:16.262370    2567 generic.go:356] "Generic (PLEG): container finished" podID="0130260a-d6c1-4b98-81d9-0ee4554b0a5a" containerID="d1fd737280f68bc5993dde3b7257c4733dba2faf0eb8005641aeabe79f0fcf05" exitCode=0
Sep 17 17:52:16.262504 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:52:16.262405    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-76857dd9b4-cvqt8" event={"ID":"0130260a-d6c1-4b98-81d9-0ee4554b0a5a","Type":"ContainerDied","Data":"d1fd737280f68bc5993dde3b7257c4733dba2faf0eb8005641aeabe79f0fcf05"}
Sep 17 17:52:16.262504 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:52:16.262449    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-76857dd9b4-cvqt8" event={"ID":"0130260a-d6c1-4b98-81d9-0ee4554b0a5a","Type":"ContainerDied","Data":"ecf83a1446f5237671e46d0eb0ec29bb50a84fae95e7f6be534914847bfdccf3"}
Sep 17 17:52:16.262504 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:52:16.262469    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-76857dd9b4-cvqt8"
Sep 17 17:52:16.267141 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:52:16.267125    2567 scope.go:117] "RemoveContainer" containerID="ab30fee80574d6269309bf06605b2ecb0de302fbeb6fe08a62a36443ec1dbece"
Sep 17 17:52:16.277259 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:52:16.277243    2567 scope.go:117] "RemoveContainer" containerID="1f212862496e96cdee944ccdeca1dd541b81a9d1a0897014f3d155696f0a97c2"
Sep 17 17:52:16.285856 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:52:16.285832    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-xzhr5"]
Sep 17 17:52:16.286358 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:52:16.286340    2567 scope.go:117] "RemoveContainer" containerID="7327999b4426e99567e59a7d1f82ca7093aa838e94de41a7b2d9cd45dfc85810"
Sep 17 17:52:16.286645 ip-10-0-139-237 kubenswrapper[2567]: E0917 17:52:16.286626    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7327999b4426e99567e59a7d1f82ca7093aa838e94de41a7b2d9cd45dfc85810\": container with ID starting with 7327999b4426e99567e59a7d1f82ca7093aa838e94de41a7b2d9cd45dfc85810 not found: ID does not exist" containerID="7327999b4426e99567e59a7d1f82ca7093aa838e94de41a7b2d9cd45dfc85810"
Sep 17 17:52:16.286717 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:52:16.286652    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7327999b4426e99567e59a7d1f82ca7093aa838e94de41a7b2d9cd45dfc85810"} err="failed to get container status \"7327999b4426e99567e59a7d1f82ca7093aa838e94de41a7b2d9cd45dfc85810\": rpc error: code = NotFound desc = could not find container \"7327999b4426e99567e59a7d1f82ca7093aa838e94de41a7b2d9cd45dfc85810\": container with ID starting with 7327999b4426e99567e59a7d1f82ca7093aa838e94de41a7b2d9cd45dfc85810 not found: ID does not exist"
Sep 17 17:52:16.286717 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:52:16.286693    2567 scope.go:117] "RemoveContainer" containerID="ab30fee80574d6269309bf06605b2ecb0de302fbeb6fe08a62a36443ec1dbece"
Sep 17 17:52:16.287005 ip-10-0-139-237 kubenswrapper[2567]: E0917 17:52:16.286983    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab30fee80574d6269309bf06605b2ecb0de302fbeb6fe08a62a36443ec1dbece\": container with ID starting with ab30fee80574d6269309bf06605b2ecb0de302fbeb6fe08a62a36443ec1dbece not found: ID does not exist" containerID="ab30fee80574d6269309bf06605b2ecb0de302fbeb6fe08a62a36443ec1dbece"
Sep 17 17:52:16.287096 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:52:16.287009    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab30fee80574d6269309bf06605b2ecb0de302fbeb6fe08a62a36443ec1dbece"} err="failed to get container status \"ab30fee80574d6269309bf06605b2ecb0de302fbeb6fe08a62a36443ec1dbece\": rpc error: code = NotFound desc = could not find container \"ab30fee80574d6269309bf06605b2ecb0de302fbeb6fe08a62a36443ec1dbece\": container with ID starting with ab30fee80574d6269309bf06605b2ecb0de302fbeb6fe08a62a36443ec1dbece not found: ID does not exist"
Sep 17 17:52:16.287096 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:52:16.287028    2567 scope.go:117] "RemoveContainer" containerID="1f212862496e96cdee944ccdeca1dd541b81a9d1a0897014f3d155696f0a97c2"
Sep 17 17:52:16.287348 ip-10-0-139-237 kubenswrapper[2567]: E0917 17:52:16.287318    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f212862496e96cdee944ccdeca1dd541b81a9d1a0897014f3d155696f0a97c2\": container with ID starting with 1f212862496e96cdee944ccdeca1dd541b81a9d1a0897014f3d155696f0a97c2 not found: ID does not exist" containerID="1f212862496e96cdee944ccdeca1dd541b81a9d1a0897014f3d155696f0a97c2"
Sep 17 17:52:16.287472 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:52:16.287359    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f212862496e96cdee944ccdeca1dd541b81a9d1a0897014f3d155696f0a97c2"} err="failed to get container status \"1f212862496e96cdee944ccdeca1dd541b81a9d1a0897014f3d155696f0a97c2\": rpc error: code = NotFound desc = could not find container \"1f212862496e96cdee944ccdeca1dd541b81a9d1a0897014f3d155696f0a97c2\": container with ID starting with 1f212862496e96cdee944ccdeca1dd541b81a9d1a0897014f3d155696f0a97c2 not found: ID does not exist"
Sep 17 17:52:16.287472 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:52:16.287382    2567 scope.go:117] "RemoveContainer" containerID="02fad77b74cd3a78ee4791c8b7d0cb32bcdd4b429711cb1cf139ad7dc46ed57b"
Sep 17 17:52:16.289589 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:52:16.289568    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-xzhr5"]
Sep 17 17:52:16.295503 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:52:16.295483    2567 scope.go:117] "RemoveContainer" containerID="e6d053e1149269fa0a2d9f93ac9e1b177701254add208af0bf2c90a730a679ee"
Sep 17 17:52:16.301024 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:52:16.301004    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-76857dd9b4-cvqt8"]
Sep 17 17:52:16.304757 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:52:16.304735    2567 scope.go:117] "RemoveContainer" containerID="7de5f07df5ddf975d8972815785d02c0382e3db33db2bdbf3bdc9fe24a06c4cc"
Sep 17 17:52:16.306591 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:52:16.306570    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-76857dd9b4-cvqt8"]
Sep 17 17:52:16.312774 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:52:16.312758    2567 scope.go:117] "RemoveContainer" containerID="02fad77b74cd3a78ee4791c8b7d0cb32bcdd4b429711cb1cf139ad7dc46ed57b"
Sep 17 17:52:16.313038 ip-10-0-139-237 kubenswrapper[2567]: E0917 17:52:16.313021    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"02fad77b74cd3a78ee4791c8b7d0cb32bcdd4b429711cb1cf139ad7dc46ed57b\": container with ID starting with 02fad77b74cd3a78ee4791c8b7d0cb32bcdd4b429711cb1cf139ad7dc46ed57b not found: ID does not exist" containerID="02fad77b74cd3a78ee4791c8b7d0cb32bcdd4b429711cb1cf139ad7dc46ed57b"
Sep 17 17:52:16.313119 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:52:16.313044    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"02fad77b74cd3a78ee4791c8b7d0cb32bcdd4b429711cb1cf139ad7dc46ed57b"} err="failed to get container status \"02fad77b74cd3a78ee4791c8b7d0cb32bcdd4b429711cb1cf139ad7dc46ed57b\": rpc error: code = NotFound desc = could not find container \"02fad77b74cd3a78ee4791c8b7d0cb32bcdd4b429711cb1cf139ad7dc46ed57b\": container with ID starting with 02fad77b74cd3a78ee4791c8b7d0cb32bcdd4b429711cb1cf139ad7dc46ed57b not found: ID does not exist"
Sep 17 17:52:16.313119 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:52:16.313062    2567 scope.go:117] "RemoveContainer" containerID="e6d053e1149269fa0a2d9f93ac9e1b177701254add208af0bf2c90a730a679ee"
Sep 17 17:52:16.313368 ip-10-0-139-237 kubenswrapper[2567]: E0917 17:52:16.313344    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e6d053e1149269fa0a2d9f93ac9e1b177701254add208af0bf2c90a730a679ee\": container with ID starting with e6d053e1149269fa0a2d9f93ac9e1b177701254add208af0bf2c90a730a679ee not found: ID does not exist" containerID="e6d053e1149269fa0a2d9f93ac9e1b177701254add208af0bf2c90a730a679ee"
Sep 17 17:52:16.313414 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:52:16.313377    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e6d053e1149269fa0a2d9f93ac9e1b177701254add208af0bf2c90a730a679ee"} err="failed to get container status \"e6d053e1149269fa0a2d9f93ac9e1b177701254add208af0bf2c90a730a679ee\": rpc error: code = NotFound desc = could not find container \"e6d053e1149269fa0a2d9f93ac9e1b177701254add208af0bf2c90a730a679ee\": container with ID starting with e6d053e1149269fa0a2d9f93ac9e1b177701254add208af0bf2c90a730a679ee not found: ID does not exist"
Sep 17 17:52:16.313414 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:52:16.313400    2567 scope.go:117] "RemoveContainer" containerID="7de5f07df5ddf975d8972815785d02c0382e3db33db2bdbf3bdc9fe24a06c4cc"
Sep 17 17:52:16.313717 ip-10-0-139-237 kubenswrapper[2567]: E0917 17:52:16.313696    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7de5f07df5ddf975d8972815785d02c0382e3db33db2bdbf3bdc9fe24a06c4cc\": container with ID starting with 7de5f07df5ddf975d8972815785d02c0382e3db33db2bdbf3bdc9fe24a06c4cc not found: ID does not exist" containerID="7de5f07df5ddf975d8972815785d02c0382e3db33db2bdbf3bdc9fe24a06c4cc"
Sep 17 17:52:16.313811 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:52:16.313723    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7de5f07df5ddf975d8972815785d02c0382e3db33db2bdbf3bdc9fe24a06c4cc"} err="failed to get container status \"7de5f07df5ddf975d8972815785d02c0382e3db33db2bdbf3bdc9fe24a06c4cc\": rpc error: code = NotFound desc = could not find container \"7de5f07df5ddf975d8972815785d02c0382e3db33db2bdbf3bdc9fe24a06c4cc\": container with ID starting with 7de5f07df5ddf975d8972815785d02c0382e3db33db2bdbf3bdc9fe24a06c4cc not found: ID does not exist"
Sep 17 17:52:16.313811 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:52:16.313743    2567 scope.go:117] "RemoveContainer" containerID="2d9d5fbe3dcdc35839cf4baafc69a129ba230c85142ff895333f0bb4fbefff2d"
Sep 17 17:52:16.317718 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:52:16.317632    2567 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/496d8e39-534e-4318-be60-0f872782e0c4-proxy-tls\") on node \"ip-10-0-139-237.ec2.internal\" DevicePath \"\""
Sep 17 17:52:16.317718 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:52:16.317656    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m87s8\" (UniqueName: \"kubernetes.io/projected/496d8e39-534e-4318-be60-0f872782e0c4-kube-api-access-m87s8\") on node \"ip-10-0-139-237.ec2.internal\" DevicePath \"\""
Sep 17 17:52:16.319675 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:52:16.319651    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-74fc5cfbf9-4bx9x"]
Sep 17 17:52:16.322751 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:52:16.322733    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-74fc5cfbf9-4bx9x"]
Sep 17 17:52:16.322833 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:52:16.322738    2567 scope.go:117] "RemoveContainer" containerID="d1fd737280f68bc5993dde3b7257c4733dba2faf0eb8005641aeabe79f0fcf05"
Sep 17 17:52:16.330157 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:52:16.330143    2567 scope.go:117] "RemoveContainer" containerID="251c6264618654049ff9979487e9296c6168fbd5a0c3db6ffbc5354424098227"
Sep 17 17:52:16.337412 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:52:16.337382    2567 scope.go:117] "RemoveContainer" containerID="2d9d5fbe3dcdc35839cf4baafc69a129ba230c85142ff895333f0bb4fbefff2d"
Sep 17 17:52:16.337711 ip-10-0-139-237 kubenswrapper[2567]: E0917 17:52:16.337689    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d9d5fbe3dcdc35839cf4baafc69a129ba230c85142ff895333f0bb4fbefff2d\": container with ID starting with 2d9d5fbe3dcdc35839cf4baafc69a129ba230c85142ff895333f0bb4fbefff2d not found: ID does not exist" containerID="2d9d5fbe3dcdc35839cf4baafc69a129ba230c85142ff895333f0bb4fbefff2d"
Sep 17 17:52:16.337804 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:52:16.337717    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d9d5fbe3dcdc35839cf4baafc69a129ba230c85142ff895333f0bb4fbefff2d"} err="failed to get container status \"2d9d5fbe3dcdc35839cf4baafc69a129ba230c85142ff895333f0bb4fbefff2d\": rpc error: code = NotFound desc = could not find container \"2d9d5fbe3dcdc35839cf4baafc69a129ba230c85142ff895333f0bb4fbefff2d\": container with ID starting with 2d9d5fbe3dcdc35839cf4baafc69a129ba230c85142ff895333f0bb4fbefff2d not found: ID does not exist"
Sep 17 17:52:16.337804 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:52:16.337733    2567 scope.go:117] "RemoveContainer" containerID="d1fd737280f68bc5993dde3b7257c4733dba2faf0eb8005641aeabe79f0fcf05"
Sep 17 17:52:16.337999 ip-10-0-139-237 kubenswrapper[2567]: E0917 17:52:16.337986    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d1fd737280f68bc5993dde3b7257c4733dba2faf0eb8005641aeabe79f0fcf05\": container with ID starting with d1fd737280f68bc5993dde3b7257c4733dba2faf0eb8005641aeabe79f0fcf05 not found: ID does not exist" containerID="d1fd737280f68bc5993dde3b7257c4733dba2faf0eb8005641aeabe79f0fcf05"
Sep 17 17:52:16.338039 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:52:16.338002    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1fd737280f68bc5993dde3b7257c4733dba2faf0eb8005641aeabe79f0fcf05"} err="failed to get container status \"d1fd737280f68bc5993dde3b7257c4733dba2faf0eb8005641aeabe79f0fcf05\": rpc error: code = NotFound desc = could not find container \"d1fd737280f68bc5993dde3b7257c4733dba2faf0eb8005641aeabe79f0fcf05\": container with ID starting with d1fd737280f68bc5993dde3b7257c4733dba2faf0eb8005641aeabe79f0fcf05 not found: ID does not exist"
Sep 17 17:52:16.338039 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:52:16.338014    2567 scope.go:117] "RemoveContainer" containerID="251c6264618654049ff9979487e9296c6168fbd5a0c3db6ffbc5354424098227"
Sep 17 17:52:16.338259 ip-10-0-139-237 kubenswrapper[2567]: E0917 17:52:16.338239    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"251c6264618654049ff9979487e9296c6168fbd5a0c3db6ffbc5354424098227\": container with ID starting with 251c6264618654049ff9979487e9296c6168fbd5a0c3db6ffbc5354424098227 not found: ID does not exist" containerID="251c6264618654049ff9979487e9296c6168fbd5a0c3db6ffbc5354424098227"
Sep 17 17:52:16.338308 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:52:16.338263    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"251c6264618654049ff9979487e9296c6168fbd5a0c3db6ffbc5354424098227"} err="failed to get container status \"251c6264618654049ff9979487e9296c6168fbd5a0c3db6ffbc5354424098227\": rpc error: code = NotFound desc = could not find container \"251c6264618654049ff9979487e9296c6168fbd5a0c3db6ffbc5354424098227\": container with ID starting with 251c6264618654049ff9979487e9296c6168fbd5a0c3db6ffbc5354424098227 not found: ID does not exist"
Sep 17 17:52:17.285408 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:52:17.285373    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0130260a-d6c1-4b98-81d9-0ee4554b0a5a" path="/var/lib/kubelet/pods/0130260a-d6c1-4b98-81d9-0ee4554b0a5a/volumes"
Sep 17 17:52:17.285900 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:52:17.285885    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="496d8e39-534e-4318-be60-0f872782e0c4" path="/var/lib/kubelet/pods/496d8e39-534e-4318-be60-0f872782e0c4/volumes"
Sep 17 17:52:17.286622 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:52:17.286608    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e68679b4-02fe-4048-b466-dcb4dd11da7a" path="/var/lib/kubelet/pods/e68679b4-02fe-4048-b466-dcb4dd11da7a/volumes"
Sep 17 17:52:17.349469 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:52:17.349439    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 17:52:20.254122 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:52:20.254094    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-081d8-predictor-7f479c548c-h2qk4"
Sep 17 17:52:20.254636 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:52:20.254508    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-081d8-predictor-7f479c548c-h2qk4" podUID="b92d1167-af4a-4333-bcfe-73b07e791faf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Sep 17 17:52:20.815223 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:52:20.815188    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 17:52:20.962018 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:52:20.961984    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 17:52:30.254812 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:52:30.254775    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-081d8-predictor-7f479c548c-h2qk4" podUID="b92d1167-af4a-4333-bcfe-73b07e791faf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Sep 17 17:52:40.255049 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:52:40.255008    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-081d8-predictor-7f479c548c-h2qk4" podUID="b92d1167-af4a-4333-bcfe-73b07e791faf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Sep 17 17:52:42.131555 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:52:42.131461    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 17:52:42.194661 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:52:42.194626    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 17:52:50.254932 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:52:50.254893    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-081d8-predictor-7f479c548c-h2qk4" podUID="b92d1167-af4a-4333-bcfe-73b07e791faf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Sep 17 17:53:00.255328 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:53:00.255294    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-081d8-predictor-7f479c548c-h2qk4"
Sep 17 17:53:11.159621 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:53:11.159583    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/sequence-graph-081d8-66964d46f7-ghmwg"]
Sep 17 17:53:11.160026 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:53:11.159932    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e68679b4-02fe-4048-b466-dcb4dd11da7a" containerName="kube-rbac-proxy"
Sep 17 17:53:11.160026 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:53:11.159945    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="e68679b4-02fe-4048-b466-dcb4dd11da7a" containerName="kube-rbac-proxy"
Sep 17 17:53:11.160026 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:53:11.159953    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0130260a-d6c1-4b98-81d9-0ee4554b0a5a" containerName="kube-rbac-proxy"
Sep 17 17:53:11.160026 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:53:11.159961    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="0130260a-d6c1-4b98-81d9-0ee4554b0a5a" containerName="kube-rbac-proxy"
Sep 17 17:53:11.160026 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:53:11.159975    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0130260a-d6c1-4b98-81d9-0ee4554b0a5a" containerName="storage-initializer"
Sep 17 17:53:11.160026 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:53:11.159984    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="0130260a-d6c1-4b98-81d9-0ee4554b0a5a" containerName="storage-initializer"
Sep 17 17:53:11.160026 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:53:11.159995    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e68679b4-02fe-4048-b466-dcb4dd11da7a" containerName="storage-initializer"
Sep 17 17:53:11.160026 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:53:11.160000    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="e68679b4-02fe-4048-b466-dcb4dd11da7a" containerName="storage-initializer"
Sep 17 17:53:11.160026 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:53:11.160008    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="496d8e39-534e-4318-be60-0f872782e0c4" containerName="kube-rbac-proxy"
Sep 17 17:53:11.160026 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:53:11.160013    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="496d8e39-534e-4318-be60-0f872782e0c4" containerName="kube-rbac-proxy"
Sep 17 17:53:11.160026 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:53:11.160023    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="496d8e39-534e-4318-be60-0f872782e0c4" containerName="storage-initializer"
Sep 17 17:53:11.160026 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:53:11.160028    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="496d8e39-534e-4318-be60-0f872782e0c4" containerName="storage-initializer"
Sep 17 17:53:11.160026 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:53:11.160038    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e68679b4-02fe-4048-b466-dcb4dd11da7a" containerName="kserve-container"
Sep 17 17:53:11.160444 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:53:11.160043    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="e68679b4-02fe-4048-b466-dcb4dd11da7a" containerName="kserve-container"
Sep 17 17:53:11.160444 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:53:11.160052    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0130260a-d6c1-4b98-81d9-0ee4554b0a5a" containerName="kserve-container"
Sep 17 17:53:11.160444 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:53:11.160058    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="0130260a-d6c1-4b98-81d9-0ee4554b0a5a" containerName="kserve-container"
Sep 17 17:53:11.160444 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:53:11.160066    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="496d8e39-534e-4318-be60-0f872782e0c4" containerName="kserve-container"
Sep 17 17:53:11.160444 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:53:11.160071    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="496d8e39-534e-4318-be60-0f872782e0c4" containerName="kserve-container"
Sep 17 17:53:11.160444 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:53:11.160118    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="e68679b4-02fe-4048-b466-dcb4dd11da7a" containerName="kube-rbac-proxy"
Sep 17 17:53:11.160444 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:53:11.160125    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="496d8e39-534e-4318-be60-0f872782e0c4" containerName="kube-rbac-proxy"
Sep 17 17:53:11.160444 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:53:11.160131    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="e68679b4-02fe-4048-b466-dcb4dd11da7a" containerName="kserve-container"
Sep 17 17:53:11.160444 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:53:11.160138    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="496d8e39-534e-4318-be60-0f872782e0c4" containerName="kserve-container"
Sep 17 17:53:11.160444 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:53:11.160144    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="0130260a-d6c1-4b98-81d9-0ee4554b0a5a" containerName="kserve-container"
Sep 17 17:53:11.160444 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:53:11.160150    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="0130260a-d6c1-4b98-81d9-0ee4554b0a5a" containerName="kube-rbac-proxy"
Sep 17 17:53:11.161110 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:53:11.161078    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 17:53:11.164584 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:53:11.164566    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-081d8-66964d46f7-ghmwg"
Sep 17 17:53:11.167291 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:53:11.167271    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-081d8-kube-rbac-proxy-sar-config\""
Sep 17 17:53:11.167394 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:53:11.167304    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-081d8-serving-cert\""
Sep 17 17:53:11.172792 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:53:11.172773    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-081d8-66964d46f7-ghmwg"]
Sep 17 17:53:11.282724 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:53:11.282693    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/976221a4-d467-49c4-9b8b-174b8fba34e3-openshift-service-ca-bundle\") pod \"sequence-graph-081d8-66964d46f7-ghmwg\" (UID: \"976221a4-d467-49c4-9b8b-174b8fba34e3\") " pod="kserve-ci-e2e-test/sequence-graph-081d8-66964d46f7-ghmwg"
Sep 17 17:53:11.282910 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:53:11.282733    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/976221a4-d467-49c4-9b8b-174b8fba34e3-proxy-tls\") pod \"sequence-graph-081d8-66964d46f7-ghmwg\" (UID: \"976221a4-d467-49c4-9b8b-174b8fba34e3\") " pod="kserve-ci-e2e-test/sequence-graph-081d8-66964d46f7-ghmwg"
Sep 17 17:53:11.383490 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:53:11.383451    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/976221a4-d467-49c4-9b8b-174b8fba34e3-openshift-service-ca-bundle\") pod \"sequence-graph-081d8-66964d46f7-ghmwg\" (UID: \"976221a4-d467-49c4-9b8b-174b8fba34e3\") " pod="kserve-ci-e2e-test/sequence-graph-081d8-66964d46f7-ghmwg"
Sep 17 17:53:11.383490 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:53:11.383495    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/976221a4-d467-49c4-9b8b-174b8fba34e3-proxy-tls\") pod \"sequence-graph-081d8-66964d46f7-ghmwg\" (UID: \"976221a4-d467-49c4-9b8b-174b8fba34e3\") " pod="kserve-ci-e2e-test/sequence-graph-081d8-66964d46f7-ghmwg"
Sep 17 17:53:11.384077 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:53:11.384053    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/976221a4-d467-49c4-9b8b-174b8fba34e3-openshift-service-ca-bundle\") pod \"sequence-graph-081d8-66964d46f7-ghmwg\" (UID: \"976221a4-d467-49c4-9b8b-174b8fba34e3\") " pod="kserve-ci-e2e-test/sequence-graph-081d8-66964d46f7-ghmwg"
Sep 17 17:53:11.385979 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:53:11.385948    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/976221a4-d467-49c4-9b8b-174b8fba34e3-proxy-tls\") pod \"sequence-graph-081d8-66964d46f7-ghmwg\" (UID: \"976221a4-d467-49c4-9b8b-174b8fba34e3\") " pod="kserve-ci-e2e-test/sequence-graph-081d8-66964d46f7-ghmwg"
Sep 17 17:53:11.474961 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:53:11.474899    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-081d8-66964d46f7-ghmwg"
Sep 17 17:53:11.479354 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:53:11.479328    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 17:53:11.746480 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:53:11.746322    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-081d8-66964d46f7-ghmwg"]
Sep 17 17:53:11.749249 ip-10-0-139-237 kubenswrapper[2567]: W0917 17:53:11.749223    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod976221a4_d467_49c4_9b8b_174b8fba34e3.slice/crio-d2d4b7e8d53870bffdb563df5ff76924c059d1a947d7355430bfd50c2a5aa8d6 WatchSource:0}: Error finding container d2d4b7e8d53870bffdb563df5ff76924c059d1a947d7355430bfd50c2a5aa8d6: Status 404 returned error can't find the container with id d2d4b7e8d53870bffdb563df5ff76924c059d1a947d7355430bfd50c2a5aa8d6
Sep 17 17:53:11.754112 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:53:11.754094    2567 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 17:53:12.489133 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:53:12.489096    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-081d8-66964d46f7-ghmwg" event={"ID":"976221a4-d467-49c4-9b8b-174b8fba34e3","Type":"ContainerStarted","Data":"d2d4b7e8d53870bffdb563df5ff76924c059d1a947d7355430bfd50c2a5aa8d6"}
Sep 17 17:53:14.498698 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:53:14.498662    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-081d8-66964d46f7-ghmwg" event={"ID":"976221a4-d467-49c4-9b8b-174b8fba34e3","Type":"ContainerStarted","Data":"04532fda5677a9ae9e4e1c86e61fc95d13e3d2bddc659911114eccd5b42d056d"}
Sep 17 17:53:14.499191 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:53:14.498762    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-081d8-66964d46f7-ghmwg"
Sep 17 17:53:14.527346 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:53:14.527294    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/sequence-graph-081d8-66964d46f7-ghmwg" podStartSLOduration=1.685941191 podStartE2EDuration="3.527280199s" totalImagesPullingTime="1.841339008s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:53:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:53:14.526385563 +0000 UTC m=+735.549990722" watchObservedRunningTime="2026-09-17 17:53:14.527280199 +0000 UTC m=+735.550885334"
Sep 17 17:53:20.511055 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:53:20.511026    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/sequence-graph-081d8-66964d46f7-ghmwg"
Sep 17 17:54:22.882986 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:54:22.882911    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 17:56:52.309513 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:56:52.309481    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 17:58:22.106609 ip-10-0-139-237 kubenswrapper[2567]: I0917 17:58:22.106568    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 18:00:13.333934 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:00:13.333861    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 18:00:35.619093 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:00:35.619061    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 18:00:35.733325 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:00:35.733296    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 18:00:35.807383 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:00:35.807350    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 18:00:35.853759 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:00:35.853731    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-8592d-predictor-94446b57c-nnlbx"]
Sep 17 18:00:35.853985 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:00:35.853962    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-8592d-predictor-94446b57c-nnlbx" podUID="fb8979a8-6a3e-47e6-9c2e-9f59a659a84e" containerName="kserve-container" containerID="cri-o://3fab16144dd478b86aeb537ef3a198d1440239226d9445ad5ffa7e8c7d13e416" gracePeriod=30
Sep 17 18:00:35.854044 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:00:35.853994    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-8592d-predictor-94446b57c-nnlbx" podUID="fb8979a8-6a3e-47e6-9c2e-9f59a659a84e" containerName="kube-rbac-proxy" containerID="cri-o://1285cb2af78097681eda517bf1a3bb962e0c27967ad434e579f8975ae301458d" gracePeriod=30
Sep 17 18:00:35.908073 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:00:35.908013    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 18:00:35.959303 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:00:35.959276    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-0dcf6-predictor-5d98597747-kxs58"]
Sep 17 18:00:35.961722 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:00:35.961702    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-0dcf6-predictor-5d98597747-kxs58"
Sep 17 18:00:35.965158 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:00:35.965140    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-0dcf6-predictor-serving-cert\""
Sep 17 18:00:35.965584 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:00:35.965566    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-0dcf6-kube-rbac-proxy-sar-config\""
Sep 17 18:00:35.976052 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:00:35.976028    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-0dcf6-predictor-5d98597747-kxs58"]
Sep 17 18:00:36.002882 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:00:36.002855    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 18:00:36.028040 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:00:36.028020    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/43e1e374-0ac1-4fa7-ab65-95ed213ba4ba-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-0dcf6-predictor-5d98597747-kxs58\" (UID: \"43e1e374-0ac1-4fa7-ab65-95ed213ba4ba\") " pod="kserve-ci-e2e-test/error-404-isvc-0dcf6-predictor-5d98597747-kxs58"
Sep 17 18:00:36.028140 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:00:36.028065    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/43e1e374-0ac1-4fa7-ab65-95ed213ba4ba-proxy-tls\") pod \"error-404-isvc-0dcf6-predictor-5d98597747-kxs58\" (UID: \"43e1e374-0ac1-4fa7-ab65-95ed213ba4ba\") " pod="kserve-ci-e2e-test/error-404-isvc-0dcf6-predictor-5d98597747-kxs58"
Sep 17 18:00:36.028140 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:00:36.028123    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4nr5r\" (UniqueName: \"kubernetes.io/projected/43e1e374-0ac1-4fa7-ab65-95ed213ba4ba-kube-api-access-4nr5r\") pod \"error-404-isvc-0dcf6-predictor-5d98597747-kxs58\" (UID: \"43e1e374-0ac1-4fa7-ab65-95ed213ba4ba\") " pod="kserve-ci-e2e-test/error-404-isvc-0dcf6-predictor-5d98597747-kxs58"
Sep 17 18:00:36.069517 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:00:36.069482    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 18:00:36.129228 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:00:36.129205    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/43e1e374-0ac1-4fa7-ab65-95ed213ba4ba-proxy-tls\") pod \"error-404-isvc-0dcf6-predictor-5d98597747-kxs58\" (UID: \"43e1e374-0ac1-4fa7-ab65-95ed213ba4ba\") " pod="kserve-ci-e2e-test/error-404-isvc-0dcf6-predictor-5d98597747-kxs58"
Sep 17 18:00:36.129314 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:00:36.129254    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4nr5r\" (UniqueName: \"kubernetes.io/projected/43e1e374-0ac1-4fa7-ab65-95ed213ba4ba-kube-api-access-4nr5r\") pod \"error-404-isvc-0dcf6-predictor-5d98597747-kxs58\" (UID: \"43e1e374-0ac1-4fa7-ab65-95ed213ba4ba\") " pod="kserve-ci-e2e-test/error-404-isvc-0dcf6-predictor-5d98597747-kxs58"
Sep 17 18:00:36.129368 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:00:36.129353    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/43e1e374-0ac1-4fa7-ab65-95ed213ba4ba-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-0dcf6-predictor-5d98597747-kxs58\" (UID: \"43e1e374-0ac1-4fa7-ab65-95ed213ba4ba\") " pod="kserve-ci-e2e-test/error-404-isvc-0dcf6-predictor-5d98597747-kxs58"
Sep 17 18:00:36.129946 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:00:36.129926    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/43e1e374-0ac1-4fa7-ab65-95ed213ba4ba-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-0dcf6-predictor-5d98597747-kxs58\" (UID: \"43e1e374-0ac1-4fa7-ab65-95ed213ba4ba\") " pod="kserve-ci-e2e-test/error-404-isvc-0dcf6-predictor-5d98597747-kxs58"
Sep 17 18:00:36.131547 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:00:36.131518    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/43e1e374-0ac1-4fa7-ab65-95ed213ba4ba-proxy-tls\") pod \"error-404-isvc-0dcf6-predictor-5d98597747-kxs58\" (UID: \"43e1e374-0ac1-4fa7-ab65-95ed213ba4ba\") " pod="kserve-ci-e2e-test/error-404-isvc-0dcf6-predictor-5d98597747-kxs58"
Sep 17 18:00:36.138087 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:00:36.138068    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4nr5r\" (UniqueName: \"kubernetes.io/projected/43e1e374-0ac1-4fa7-ab65-95ed213ba4ba-kube-api-access-4nr5r\") pod \"error-404-isvc-0dcf6-predictor-5d98597747-kxs58\" (UID: \"43e1e374-0ac1-4fa7-ab65-95ed213ba4ba\") " pod="kserve-ci-e2e-test/error-404-isvc-0dcf6-predictor-5d98597747-kxs58"
Sep 17 18:00:36.252821 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:00:36.252744    2567 generic.go:356] "Generic (PLEG): container finished" podID="fb8979a8-6a3e-47e6-9c2e-9f59a659a84e" containerID="1285cb2af78097681eda517bf1a3bb962e0c27967ad434e579f8975ae301458d" exitCode=2
Sep 17 18:00:36.252959 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:00:36.252826    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-8592d-predictor-94446b57c-nnlbx" event={"ID":"fb8979a8-6a3e-47e6-9c2e-9f59a659a84e","Type":"ContainerDied","Data":"1285cb2af78097681eda517bf1a3bb962e0c27967ad434e579f8975ae301458d"}
Sep 17 18:00:36.272978 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:00:36.272955    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-0dcf6-predictor-5d98597747-kxs58"
Sep 17 18:00:36.544835 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:00:36.544801    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-0dcf6-predictor-5d98597747-kxs58"]
Sep 17 18:00:36.546983 ip-10-0-139-237 kubenswrapper[2567]: W0917 18:00:36.546960    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43e1e374_0ac1_4fa7_ab65_95ed213ba4ba.slice/crio-a1902d5a69c8bb887331df0789828cbcc8d9491226f372aff53a5cb8e9f4acb2 WatchSource:0}: Error finding container a1902d5a69c8bb887331df0789828cbcc8d9491226f372aff53a5cb8e9f4acb2: Status 404 returned error can't find the container with id a1902d5a69c8bb887331df0789828cbcc8d9491226f372aff53a5cb8e9f4acb2
Sep 17 18:00:37.260397 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:00:37.260364    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-0dcf6-predictor-5d98597747-kxs58" event={"ID":"43e1e374-0ac1-4fa7-ab65-95ed213ba4ba","Type":"ContainerStarted","Data":"e508bf3da470d99aecece89b6623d082eb856945fcae0b51d54cecfefbc86914"}
Sep 17 18:00:37.260813 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:00:37.260458    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-0dcf6-predictor-5d98597747-kxs58" event={"ID":"43e1e374-0ac1-4fa7-ab65-95ed213ba4ba","Type":"ContainerStarted","Data":"8607b90d2dd8f7fd887a7d95bd42d7abba4e204f4ca994a6fd91341865aa2d6c"}
Sep 17 18:00:37.260813 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:00:37.260492    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-0dcf6-predictor-5d98597747-kxs58" event={"ID":"43e1e374-0ac1-4fa7-ab65-95ed213ba4ba","Type":"ContainerStarted","Data":"a1902d5a69c8bb887331df0789828cbcc8d9491226f372aff53a5cb8e9f4acb2"}
Sep 17 18:00:37.280909 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:00:37.280863    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-0dcf6-predictor-5d98597747-kxs58" podStartSLOduration=2.280848887 podStartE2EDuration="2.280848887s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:00:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:00:37.279609813 +0000 UTC m=+1178.303214961" watchObservedRunningTime="2026-09-17 18:00:37.280848887 +0000 UTC m=+1178.304454022"
Sep 17 18:00:38.021627 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:00:38.021582    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-8592d-predictor-94446b57c-nnlbx" podUID="fb8979a8-6a3e-47e6-9c2e-9f59a659a84e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.29:8643/healthz\": dial tcp 10.134.0.29:8643: connect: connection refused"
Sep 17 18:00:38.027131 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:00:38.027104    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-8592d-predictor-94446b57c-nnlbx" podUID="fb8979a8-6a3e-47e6-9c2e-9f59a659a84e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Sep 17 18:00:38.265030 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:00:38.264998    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-0dcf6-predictor-5d98597747-kxs58"
Sep 17 18:00:38.265467 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:00:38.265057    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-0dcf6-predictor-5d98597747-kxs58"
Sep 17 18:00:38.266326 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:00:38.266299    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-0dcf6-predictor-5d98597747-kxs58" podUID="43e1e374-0ac1-4fa7-ab65-95ed213ba4ba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Sep 17 18:00:38.789846 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:00:38.789822    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-8592d-predictor-94446b57c-nnlbx"
Sep 17 18:00:38.849179 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:00:38.849148    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fb8979a8-6a3e-47e6-9c2e-9f59a659a84e-kube-rbac-proxy-sar-config\") pod \"fb8979a8-6a3e-47e6-9c2e-9f59a659a84e\" (UID: \"fb8979a8-6a3e-47e6-9c2e-9f59a659a84e\") "
Sep 17 18:00:38.849380 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:00:38.849196    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fb8979a8-6a3e-47e6-9c2e-9f59a659a84e-proxy-tls\") pod \"fb8979a8-6a3e-47e6-9c2e-9f59a659a84e\" (UID: \"fb8979a8-6a3e-47e6-9c2e-9f59a659a84e\") "
Sep 17 18:00:38.849617 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:00:38.849591    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb8979a8-6a3e-47e6-9c2e-9f59a659a84e-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "fb8979a8-6a3e-47e6-9c2e-9f59a659a84e" (UID: "fb8979a8-6a3e-47e6-9c2e-9f59a659a84e"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:00:38.851280 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:00:38.851261    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb8979a8-6a3e-47e6-9c2e-9f59a659a84e-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "fb8979a8-6a3e-47e6-9c2e-9f59a659a84e" (UID: "fb8979a8-6a3e-47e6-9c2e-9f59a659a84e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:00:38.949810 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:00:38.949728    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8k6q6\" (UniqueName: \"kubernetes.io/projected/fb8979a8-6a3e-47e6-9c2e-9f59a659a84e-kube-api-access-8k6q6\") pod \"fb8979a8-6a3e-47e6-9c2e-9f59a659a84e\" (UID: \"fb8979a8-6a3e-47e6-9c2e-9f59a659a84e\") "
Sep 17 18:00:38.949951 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:00:38.949943    2567 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fb8979a8-6a3e-47e6-9c2e-9f59a659a84e-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-237.ec2.internal\" DevicePath \"\""
Sep 17 18:00:38.949991 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:00:38.949958    2567 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fb8979a8-6a3e-47e6-9c2e-9f59a659a84e-proxy-tls\") on node \"ip-10-0-139-237.ec2.internal\" DevicePath \"\""
Sep 17 18:00:38.951749 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:00:38.951718    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb8979a8-6a3e-47e6-9c2e-9f59a659a84e-kube-api-access-8k6q6" (OuterVolumeSpecName: "kube-api-access-8k6q6") pod "fb8979a8-6a3e-47e6-9c2e-9f59a659a84e" (UID: "fb8979a8-6a3e-47e6-9c2e-9f59a659a84e"). InnerVolumeSpecName "kube-api-access-8k6q6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:00:39.050719 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:00:39.050685    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8k6q6\" (UniqueName: \"kubernetes.io/projected/fb8979a8-6a3e-47e6-9c2e-9f59a659a84e-kube-api-access-8k6q6\") on node \"ip-10-0-139-237.ec2.internal\" DevicePath \"\""
Sep 17 18:00:39.270362 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:00:39.270281    2567 generic.go:356] "Generic (PLEG): container finished" podID="fb8979a8-6a3e-47e6-9c2e-9f59a659a84e" containerID="3fab16144dd478b86aeb537ef3a198d1440239226d9445ad5ffa7e8c7d13e416" exitCode=0
Sep 17 18:00:39.270362 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:00:39.270342    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-8592d-predictor-94446b57c-nnlbx"
Sep 17 18:00:39.270805 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:00:39.270376    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-8592d-predictor-94446b57c-nnlbx" event={"ID":"fb8979a8-6a3e-47e6-9c2e-9f59a659a84e","Type":"ContainerDied","Data":"3fab16144dd478b86aeb537ef3a198d1440239226d9445ad5ffa7e8c7d13e416"}
Sep 17 18:00:39.270805 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:00:39.270439    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-8592d-predictor-94446b57c-nnlbx" event={"ID":"fb8979a8-6a3e-47e6-9c2e-9f59a659a84e","Type":"ContainerDied","Data":"d886469da838904c9e6f65002390cc3f08570381efcf9df0370df5db56361d80"}
Sep 17 18:00:39.270805 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:00:39.270471    2567 scope.go:117] "RemoveContainer" containerID="1285cb2af78097681eda517bf1a3bb962e0c27967ad434e579f8975ae301458d"
Sep 17 18:00:39.270805 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:00:39.270787    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-0dcf6-predictor-5d98597747-kxs58" podUID="43e1e374-0ac1-4fa7-ab65-95ed213ba4ba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Sep 17 18:00:39.279054 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:00:39.279040    2567 scope.go:117] "RemoveContainer" containerID="3fab16144dd478b86aeb537ef3a198d1440239226d9445ad5ffa7e8c7d13e416"
Sep 17 18:00:39.286893 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:00:39.286873    2567 scope.go:117] "RemoveContainer" containerID="1285cb2af78097681eda517bf1a3bb962e0c27967ad434e579f8975ae301458d"
Sep 17 18:00:39.287117 ip-10-0-139-237 kubenswrapper[2567]: E0917 18:00:39.287100    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1285cb2af78097681eda517bf1a3bb962e0c27967ad434e579f8975ae301458d\": container with ID starting with 1285cb2af78097681eda517bf1a3bb962e0c27967ad434e579f8975ae301458d not found: ID does not exist" containerID="1285cb2af78097681eda517bf1a3bb962e0c27967ad434e579f8975ae301458d"
Sep 17 18:00:39.287176 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:00:39.287125    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1285cb2af78097681eda517bf1a3bb962e0c27967ad434e579f8975ae301458d"} err="failed to get container status \"1285cb2af78097681eda517bf1a3bb962e0c27967ad434e579f8975ae301458d\": rpc error: code = NotFound desc = could not find container \"1285cb2af78097681eda517bf1a3bb962e0c27967ad434e579f8975ae301458d\": container with ID starting with 1285cb2af78097681eda517bf1a3bb962e0c27967ad434e579f8975ae301458d not found: ID does not exist"
Sep 17 18:00:39.287176 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:00:39.287140    2567 scope.go:117] "RemoveContainer" containerID="3fab16144dd478b86aeb537ef3a198d1440239226d9445ad5ffa7e8c7d13e416"
Sep 17 18:00:39.287400 ip-10-0-139-237 kubenswrapper[2567]: E0917 18:00:39.287381    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3fab16144dd478b86aeb537ef3a198d1440239226d9445ad5ffa7e8c7d13e416\": container with ID starting with 3fab16144dd478b86aeb537ef3a198d1440239226d9445ad5ffa7e8c7d13e416 not found: ID does not exist" containerID="3fab16144dd478b86aeb537ef3a198d1440239226d9445ad5ffa7e8c7d13e416"
Sep 17 18:00:39.287451 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:00:39.287406    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3fab16144dd478b86aeb537ef3a198d1440239226d9445ad5ffa7e8c7d13e416"} err="failed to get container status \"3fab16144dd478b86aeb537ef3a198d1440239226d9445ad5ffa7e8c7d13e416\": rpc error: code = NotFound desc = could not find container \"3fab16144dd478b86aeb537ef3a198d1440239226d9445ad5ffa7e8c7d13e416\": container with ID starting with 3fab16144dd478b86aeb537ef3a198d1440239226d9445ad5ffa7e8c7d13e416 not found: ID does not exist"
Sep 17 18:00:39.293594 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:00:39.293572    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-8592d-predictor-94446b57c-nnlbx"]
Sep 17 18:00:39.299895 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:00:39.299875    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-8592d-predictor-94446b57c-nnlbx"]
Sep 17 18:00:41.285309 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:00:41.285278    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fb8979a8-6a3e-47e6-9c2e-9f59a659a84e" path="/var/lib/kubelet/pods/fb8979a8-6a3e-47e6-9c2e-9f59a659a84e/volumes"
Sep 17 18:00:44.274839 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:00:44.274808    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-0dcf6-predictor-5d98597747-kxs58"
Sep 17 18:00:44.275288 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:00:44.275206    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-0dcf6-predictor-5d98597747-kxs58" podUID="43e1e374-0ac1-4fa7-ab65-95ed213ba4ba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Sep 17 18:00:54.275355 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:00:54.275314    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-0dcf6-predictor-5d98597747-kxs58" podUID="43e1e374-0ac1-4fa7-ab65-95ed213ba4ba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Sep 17 18:01:04.275411 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:01:04.275374    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-0dcf6-predictor-5d98597747-kxs58" podUID="43e1e374-0ac1-4fa7-ab65-95ed213ba4ba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Sep 17 18:01:14.275828 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:01:14.275788    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-0dcf6-predictor-5d98597747-kxs58" podUID="43e1e374-0ac1-4fa7-ab65-95ed213ba4ba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Sep 17 18:01:24.276344 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:01:24.276312    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-0dcf6-predictor-5d98597747-kxs58"
Sep 17 18:01:25.789810 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:01:25.789775    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-081d8-66964d46f7-ghmwg"]
Sep 17 18:01:25.790232 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:01:25.789985    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/sequence-graph-081d8-66964d46f7-ghmwg" podUID="976221a4-d467-49c4-9b8b-174b8fba34e3" containerName="sequence-graph-081d8" containerID="cri-o://04532fda5677a9ae9e4e1c86e61fc95d13e3d2bddc659911114eccd5b42d056d" gracePeriod=30
Sep 17 18:01:25.827004 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:01:25.826970    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 18:01:25.943519 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:01:25.943483    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 18:01:25.961935 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:01:25.961899    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-081d8-predictor-7f479c548c-h2qk4"]
Sep 17 18:01:25.962212 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:01:25.962184    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-081d8-predictor-7f479c548c-h2qk4" podUID="b92d1167-af4a-4333-bcfe-73b07e791faf" containerName="kserve-container" containerID="cri-o://0dbb7b93cb833b6517a2fdae35aee1a473d69e9a3f58094bd6e409359fc9d8f9" gracePeriod=30
Sep 17 18:01:25.962298 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:01:25.962246    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-081d8-predictor-7f479c548c-h2qk4" podUID="b92d1167-af4a-4333-bcfe-73b07e791faf" containerName="kube-rbac-proxy" containerID="cri-o://cbaa9fb3c40255bf1b4486dde27a4db45aceb4a46951120e7f2de876d025c5b6" gracePeriod=30
Sep 17 18:01:26.041696 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:01:26.041573    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 18:01:26.156129 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:01:26.156083    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 18:01:26.163823 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:01:26.163795    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-8649f-predictor-f69888d4c-dvxpd"]
Sep 17 18:01:26.164188 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:01:26.164175    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fb8979a8-6a3e-47e6-9c2e-9f59a659a84e" containerName="kube-rbac-proxy"
Sep 17 18:01:26.164239 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:01:26.164193    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="fb8979a8-6a3e-47e6-9c2e-9f59a659a84e" containerName="kube-rbac-proxy"
Sep 17 18:01:26.164239 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:01:26.164213    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fb8979a8-6a3e-47e6-9c2e-9f59a659a84e" containerName="kserve-container"
Sep 17 18:01:26.164239 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:01:26.164220    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="fb8979a8-6a3e-47e6-9c2e-9f59a659a84e" containerName="kserve-container"
Sep 17 18:01:26.164331 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:01:26.164273    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="fb8979a8-6a3e-47e6-9c2e-9f59a659a84e" containerName="kserve-container"
Sep 17 18:01:26.164331 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:01:26.164286    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="fb8979a8-6a3e-47e6-9c2e-9f59a659a84e" containerName="kube-rbac-proxy"
Sep 17 18:01:26.166151 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:01:26.166130    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-8649f-predictor-f69888d4c-dvxpd"
Sep 17 18:01:26.169371 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:01:26.169354    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-8649f-predictor-serving-cert\""
Sep 17 18:01:26.169480 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:01:26.169390    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-8649f-kube-rbac-proxy-sar-config\""
Sep 17 18:01:26.177226 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:01:26.177203    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-8649f-predictor-f69888d4c-dvxpd"]
Sep 17 18:01:26.231233 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:01:26.231193    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 18:01:26.316317 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:01:26.316206    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 18:01:26.344726 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:01:26.344697    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vkmlp\" (UniqueName: \"kubernetes.io/projected/74c1ac49-8a02-45ec-b881-59bbb009d858-kube-api-access-vkmlp\") pod \"error-404-isvc-8649f-predictor-f69888d4c-dvxpd\" (UID: \"74c1ac49-8a02-45ec-b881-59bbb009d858\") " pod="kserve-ci-e2e-test/error-404-isvc-8649f-predictor-f69888d4c-dvxpd"
Sep 17 18:01:26.344865 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:01:26.344750    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/74c1ac49-8a02-45ec-b881-59bbb009d858-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-8649f-predictor-f69888d4c-dvxpd\" (UID: \"74c1ac49-8a02-45ec-b881-59bbb009d858\") " pod="kserve-ci-e2e-test/error-404-isvc-8649f-predictor-f69888d4c-dvxpd"
Sep 17 18:01:26.344865 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:01:26.344816    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/74c1ac49-8a02-45ec-b881-59bbb009d858-proxy-tls\") pod \"error-404-isvc-8649f-predictor-f69888d4c-dvxpd\" (UID: \"74c1ac49-8a02-45ec-b881-59bbb009d858\") " pod="kserve-ci-e2e-test/error-404-isvc-8649f-predictor-f69888d4c-dvxpd"
Sep 17 18:01:26.417594 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:01:26.417563    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 18:01:26.445690 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:01:26.445665    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/74c1ac49-8a02-45ec-b881-59bbb009d858-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-8649f-predictor-f69888d4c-dvxpd\" (UID: \"74c1ac49-8a02-45ec-b881-59bbb009d858\") " pod="kserve-ci-e2e-test/error-404-isvc-8649f-predictor-f69888d4c-dvxpd"
Sep 17 18:01:26.445814 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:01:26.445720    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/74c1ac49-8a02-45ec-b881-59bbb009d858-proxy-tls\") pod \"error-404-isvc-8649f-predictor-f69888d4c-dvxpd\" (UID: \"74c1ac49-8a02-45ec-b881-59bbb009d858\") " pod="kserve-ci-e2e-test/error-404-isvc-8649f-predictor-f69888d4c-dvxpd"
Sep 17 18:01:26.445814 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:01:26.445791    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vkmlp\" (UniqueName: \"kubernetes.io/projected/74c1ac49-8a02-45ec-b881-59bbb009d858-kube-api-access-vkmlp\") pod \"error-404-isvc-8649f-predictor-f69888d4c-dvxpd\" (UID: \"74c1ac49-8a02-45ec-b881-59bbb009d858\") " pod="kserve-ci-e2e-test/error-404-isvc-8649f-predictor-f69888d4c-dvxpd"
Sep 17 18:01:26.446376 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:01:26.446346    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/74c1ac49-8a02-45ec-b881-59bbb009d858-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-8649f-predictor-f69888d4c-dvxpd\" (UID: \"74c1ac49-8a02-45ec-b881-59bbb009d858\") " pod="kserve-ci-e2e-test/error-404-isvc-8649f-predictor-f69888d4c-dvxpd"
Sep 17 18:01:26.448163 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:01:26.448141    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/74c1ac49-8a02-45ec-b881-59bbb009d858-proxy-tls\") pod \"error-404-isvc-8649f-predictor-f69888d4c-dvxpd\" (UID: \"74c1ac49-8a02-45ec-b881-59bbb009d858\") " pod="kserve-ci-e2e-test/error-404-isvc-8649f-predictor-f69888d4c-dvxpd"
Sep 17 18:01:26.454739 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:01:26.454719    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vkmlp\" (UniqueName: \"kubernetes.io/projected/74c1ac49-8a02-45ec-b881-59bbb009d858-kube-api-access-vkmlp\") pod \"error-404-isvc-8649f-predictor-f69888d4c-dvxpd\" (UID: \"74c1ac49-8a02-45ec-b881-59bbb009d858\") " pod="kserve-ci-e2e-test/error-404-isvc-8649f-predictor-f69888d4c-dvxpd"
Sep 17 18:01:26.459782 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:01:26.459758    2567 generic.go:356] "Generic (PLEG): container finished" podID="b92d1167-af4a-4333-bcfe-73b07e791faf" containerID="cbaa9fb3c40255bf1b4486dde27a4db45aceb4a46951120e7f2de876d025c5b6" exitCode=2
Sep 17 18:01:26.459883 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:01:26.459808    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-081d8-predictor-7f479c548c-h2qk4" event={"ID":"b92d1167-af4a-4333-bcfe-73b07e791faf","Type":"ContainerDied","Data":"cbaa9fb3c40255bf1b4486dde27a4db45aceb4a46951120e7f2de876d025c5b6"}
Sep 17 18:01:26.476066 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:01:26.476048    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-8649f-predictor-f69888d4c-dvxpd"
Sep 17 18:01:26.745210 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:01:26.745127    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-8649f-predictor-f69888d4c-dvxpd"]
Sep 17 18:01:26.748240 ip-10-0-139-237 kubenswrapper[2567]: W0917 18:01:26.748211    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74c1ac49_8a02_45ec_b881_59bbb009d858.slice/crio-2c3503bba228a1280d6c89cab8342efc7fb748c63f722f0bf13b1f3713ee74b9 WatchSource:0}: Error finding container 2c3503bba228a1280d6c89cab8342efc7fb748c63f722f0bf13b1f3713ee74b9: Status 404 returned error can't find the container with id 2c3503bba228a1280d6c89cab8342efc7fb748c63f722f0bf13b1f3713ee74b9
Sep 17 18:01:27.466362 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:01:27.466330    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-8649f-predictor-f69888d4c-dvxpd" event={"ID":"74c1ac49-8a02-45ec-b881-59bbb009d858","Type":"ContainerStarted","Data":"c6290751b1a7129c132628a82b06856ce94926f3149f6273c38b4d5cb8b039fa"}
Sep 17 18:01:27.466852 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:01:27.466390    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-8649f-predictor-f69888d4c-dvxpd" event={"ID":"74c1ac49-8a02-45ec-b881-59bbb009d858","Type":"ContainerStarted","Data":"848fd758bfc502873a4860495a96c1f00150c691a8c6e9d8da7d66d0a0093f16"}
Sep 17 18:01:27.466852 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:01:27.466411    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-8649f-predictor-f69888d4c-dvxpd" event={"ID":"74c1ac49-8a02-45ec-b881-59bbb009d858","Type":"ContainerStarted","Data":"2c3503bba228a1280d6c89cab8342efc7fb748c63f722f0bf13b1f3713ee74b9"}
Sep 17 18:01:27.466852 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:01:27.466467    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-8649f-predictor-f69888d4c-dvxpd"
Sep 17 18:01:27.466852 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:01:27.466489    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-8649f-predictor-f69888d4c-dvxpd"
Sep 17 18:01:27.467812 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:01:27.467786    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-8649f-predictor-f69888d4c-dvxpd" podUID="74c1ac49-8a02-45ec-b881-59bbb009d858" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Sep 17 18:01:27.486953 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:01:27.486908    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-8649f-predictor-f69888d4c-dvxpd" podStartSLOduration=1.486895958 podStartE2EDuration="1.486895958s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:01:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:01:27.484582638 +0000 UTC m=+1228.508187788" watchObservedRunningTime="2026-09-17 18:01:27.486895958 +0000 UTC m=+1228.510501093"
Sep 17 18:01:28.470165 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:01:28.470126    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-8649f-predictor-f69888d4c-dvxpd" podUID="74c1ac49-8a02-45ec-b881-59bbb009d858" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Sep 17 18:01:29.104488 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:01:29.104463    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-081d8-predictor-7f479c548c-h2qk4"
Sep 17 18:01:29.168475 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:01:29.168447    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7l9k2\" (UniqueName: \"kubernetes.io/projected/b92d1167-af4a-4333-bcfe-73b07e791faf-kube-api-access-7l9k2\") pod \"b92d1167-af4a-4333-bcfe-73b07e791faf\" (UID: \"b92d1167-af4a-4333-bcfe-73b07e791faf\") "
Sep 17 18:01:29.168658 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:01:29.168498    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b92d1167-af4a-4333-bcfe-73b07e791faf-kube-rbac-proxy-sar-config\") pod \"b92d1167-af4a-4333-bcfe-73b07e791faf\" (UID: \"b92d1167-af4a-4333-bcfe-73b07e791faf\") "
Sep 17 18:01:29.168658 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:01:29.168579    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b92d1167-af4a-4333-bcfe-73b07e791faf-proxy-tls\") pod \"b92d1167-af4a-4333-bcfe-73b07e791faf\" (UID: \"b92d1167-af4a-4333-bcfe-73b07e791faf\") "
Sep 17 18:01:29.168847 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:01:29.168823    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b92d1167-af4a-4333-bcfe-73b07e791faf-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "b92d1167-af4a-4333-bcfe-73b07e791faf" (UID: "b92d1167-af4a-4333-bcfe-73b07e791faf"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:01:29.170576 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:01:29.170499    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b92d1167-af4a-4333-bcfe-73b07e791faf-kube-api-access-7l9k2" (OuterVolumeSpecName: "kube-api-access-7l9k2") pod "b92d1167-af4a-4333-bcfe-73b07e791faf" (UID: "b92d1167-af4a-4333-bcfe-73b07e791faf"). InnerVolumeSpecName "kube-api-access-7l9k2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:01:29.170576 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:01:29.170514    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b92d1167-af4a-4333-bcfe-73b07e791faf-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "b92d1167-af4a-4333-bcfe-73b07e791faf" (UID: "b92d1167-af4a-4333-bcfe-73b07e791faf"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:01:29.269836 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:01:29.269760    2567 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b92d1167-af4a-4333-bcfe-73b07e791faf-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-237.ec2.internal\" DevicePath \"\""
Sep 17 18:01:29.269836 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:01:29.269788    2567 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b92d1167-af4a-4333-bcfe-73b07e791faf-proxy-tls\") on node \"ip-10-0-139-237.ec2.internal\" DevicePath \"\""
Sep 17 18:01:29.269836 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:01:29.269800    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7l9k2\" (UniqueName: \"kubernetes.io/projected/b92d1167-af4a-4333-bcfe-73b07e791faf-kube-api-access-7l9k2\") on node \"ip-10-0-139-237.ec2.internal\" DevicePath \"\""
Sep 17 18:01:29.475375 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:01:29.475336    2567 generic.go:356] "Generic (PLEG): container finished" podID="b92d1167-af4a-4333-bcfe-73b07e791faf" containerID="0dbb7b93cb833b6517a2fdae35aee1a473d69e9a3f58094bd6e409359fc9d8f9" exitCode=0
Sep 17 18:01:29.475375 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:01:29.475377    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-081d8-predictor-7f479c548c-h2qk4"
Sep 17 18:01:29.475949 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:01:29.475417    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-081d8-predictor-7f479c548c-h2qk4" event={"ID":"b92d1167-af4a-4333-bcfe-73b07e791faf","Type":"ContainerDied","Data":"0dbb7b93cb833b6517a2fdae35aee1a473d69e9a3f58094bd6e409359fc9d8f9"}
Sep 17 18:01:29.475949 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:01:29.475481    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-081d8-predictor-7f479c548c-h2qk4" event={"ID":"b92d1167-af4a-4333-bcfe-73b07e791faf","Type":"ContainerDied","Data":"ad162807f05a57307b60c7a24144b148b957b022b14e527a4427e761a1e9399b"}
Sep 17 18:01:29.475949 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:01:29.475509    2567 scope.go:117] "RemoveContainer" containerID="cbaa9fb3c40255bf1b4486dde27a4db45aceb4a46951120e7f2de876d025c5b6"
Sep 17 18:01:29.483920 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:01:29.483890    2567 scope.go:117] "RemoveContainer" containerID="0dbb7b93cb833b6517a2fdae35aee1a473d69e9a3f58094bd6e409359fc9d8f9"
Sep 17 18:01:29.491039 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:01:29.491018    2567 scope.go:117] "RemoveContainer" containerID="cbaa9fb3c40255bf1b4486dde27a4db45aceb4a46951120e7f2de876d025c5b6"
Sep 17 18:01:29.491470 ip-10-0-139-237 kubenswrapper[2567]: E0917 18:01:29.491434    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cbaa9fb3c40255bf1b4486dde27a4db45aceb4a46951120e7f2de876d025c5b6\": container with ID starting with cbaa9fb3c40255bf1b4486dde27a4db45aceb4a46951120e7f2de876d025c5b6 not found: ID does not exist" containerID="cbaa9fb3c40255bf1b4486dde27a4db45aceb4a46951120e7f2de876d025c5b6"
Sep 17 18:01:29.491737 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:01:29.491704    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cbaa9fb3c40255bf1b4486dde27a4db45aceb4a46951120e7f2de876d025c5b6"} err="failed to get container status \"cbaa9fb3c40255bf1b4486dde27a4db45aceb4a46951120e7f2de876d025c5b6\": rpc error: code = NotFound desc = could not find container \"cbaa9fb3c40255bf1b4486dde27a4db45aceb4a46951120e7f2de876d025c5b6\": container with ID starting with cbaa9fb3c40255bf1b4486dde27a4db45aceb4a46951120e7f2de876d025c5b6 not found: ID does not exist"
Sep 17 18:01:29.491856 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:01:29.491742    2567 scope.go:117] "RemoveContainer" containerID="0dbb7b93cb833b6517a2fdae35aee1a473d69e9a3f58094bd6e409359fc9d8f9"
Sep 17 18:01:29.492050 ip-10-0-139-237 kubenswrapper[2567]: E0917 18:01:29.492033    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0dbb7b93cb833b6517a2fdae35aee1a473d69e9a3f58094bd6e409359fc9d8f9\": container with ID starting with 0dbb7b93cb833b6517a2fdae35aee1a473d69e9a3f58094bd6e409359fc9d8f9 not found: ID does not exist" containerID="0dbb7b93cb833b6517a2fdae35aee1a473d69e9a3f58094bd6e409359fc9d8f9"
Sep 17 18:01:29.492122 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:01:29.492057    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0dbb7b93cb833b6517a2fdae35aee1a473d69e9a3f58094bd6e409359fc9d8f9"} err="failed to get container status \"0dbb7b93cb833b6517a2fdae35aee1a473d69e9a3f58094bd6e409359fc9d8f9\": rpc error: code = NotFound desc = could not find container \"0dbb7b93cb833b6517a2fdae35aee1a473d69e9a3f58094bd6e409359fc9d8f9\": container with ID starting with 0dbb7b93cb833b6517a2fdae35aee1a473d69e9a3f58094bd6e409359fc9d8f9 not found: ID does not exist"
Sep 17 18:01:29.493107 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:01:29.493087    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-081d8-predictor-7f479c548c-h2qk4"]
Sep 17 18:01:29.497272 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:01:29.497253    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-081d8-predictor-7f479c548c-h2qk4"]
Sep 17 18:01:30.509376 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:01:30.509342    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-081d8-66964d46f7-ghmwg" podUID="976221a4-d467-49c4-9b8b-174b8fba34e3" containerName="sequence-graph-081d8" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 18:01:31.286296 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:01:31.286265    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b92d1167-af4a-4333-bcfe-73b07e791faf" path="/var/lib/kubelet/pods/b92d1167-af4a-4333-bcfe-73b07e791faf/volumes"
Sep 17 18:01:33.475090 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:01:33.475061    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-8649f-predictor-f69888d4c-dvxpd"
Sep 17 18:01:33.475572 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:01:33.475523    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-8649f-predictor-f69888d4c-dvxpd" podUID="74c1ac49-8a02-45ec-b881-59bbb009d858" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Sep 17 18:01:35.509300 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:01:35.509265    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-081d8-66964d46f7-ghmwg" podUID="976221a4-d467-49c4-9b8b-174b8fba34e3" containerName="sequence-graph-081d8" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 18:01:35.804443 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:01:35.804415    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-0dcf6-84fd5d8669-w4s5w"]
Sep 17 18:01:35.804774 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:01:35.804762    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b92d1167-af4a-4333-bcfe-73b07e791faf" containerName="kube-rbac-proxy"
Sep 17 18:01:35.804827 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:01:35.804775    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="b92d1167-af4a-4333-bcfe-73b07e791faf" containerName="kube-rbac-proxy"
Sep 17 18:01:35.804827 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:01:35.804788    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b92d1167-af4a-4333-bcfe-73b07e791faf" containerName="kserve-container"
Sep 17 18:01:35.804827 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:01:35.804794    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="b92d1167-af4a-4333-bcfe-73b07e791faf" containerName="kserve-container"
Sep 17 18:01:35.804959 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:01:35.804844    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="b92d1167-af4a-4333-bcfe-73b07e791faf" containerName="kserve-container"
Sep 17 18:01:35.804959 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:01:35.804857    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="b92d1167-af4a-4333-bcfe-73b07e791faf" containerName="kube-rbac-proxy"
Sep 17 18:01:35.807759 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:01:35.807743    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-0dcf6-84fd5d8669-w4s5w"
Sep 17 18:01:35.810364 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:01:35.810339    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"ensemble-graph-0dcf6-kube-rbac-proxy-sar-config\""
Sep 17 18:01:35.810364 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:01:35.810359    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"ensemble-graph-0dcf6-serving-cert\""
Sep 17 18:01:35.817775 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:01:35.817754    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-0dcf6-84fd5d8669-w4s5w"]
Sep 17 18:01:35.818837 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:01:35.818811    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 18:01:35.921467 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:01:35.921435    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e1a3d22c-9336-478b-9229-0a4a69c76d9c-openshift-service-ca-bundle\") pod \"ensemble-graph-0dcf6-84fd5d8669-w4s5w\" (UID: \"e1a3d22c-9336-478b-9229-0a4a69c76d9c\") " pod="kserve-ci-e2e-test/ensemble-graph-0dcf6-84fd5d8669-w4s5w"
Sep 17 18:01:35.921627 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:01:35.921561    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e1a3d22c-9336-478b-9229-0a4a69c76d9c-proxy-tls\") pod \"ensemble-graph-0dcf6-84fd5d8669-w4s5w\" (UID: \"e1a3d22c-9336-478b-9229-0a4a69c76d9c\") " pod="kserve-ci-e2e-test/ensemble-graph-0dcf6-84fd5d8669-w4s5w"
Sep 17 18:01:36.022443 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:01:36.022407    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e1a3d22c-9336-478b-9229-0a4a69c76d9c-proxy-tls\") pod \"ensemble-graph-0dcf6-84fd5d8669-w4s5w\" (UID: \"e1a3d22c-9336-478b-9229-0a4a69c76d9c\") " pod="kserve-ci-e2e-test/ensemble-graph-0dcf6-84fd5d8669-w4s5w"
Sep 17 18:01:36.022642 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:01:36.022471    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e1a3d22c-9336-478b-9229-0a4a69c76d9c-openshift-service-ca-bundle\") pod \"ensemble-graph-0dcf6-84fd5d8669-w4s5w\" (UID: \"e1a3d22c-9336-478b-9229-0a4a69c76d9c\") " pod="kserve-ci-e2e-test/ensemble-graph-0dcf6-84fd5d8669-w4s5w"
Sep 17 18:01:36.022642 ip-10-0-139-237 kubenswrapper[2567]: E0917 18:01:36.022566    2567 secret.go:189] Couldn't get secret kserve-ci-e2e-test/ensemble-graph-0dcf6-serving-cert: secret "ensemble-graph-0dcf6-serving-cert" not found
Sep 17 18:01:36.022642 ip-10-0-139-237 kubenswrapper[2567]: E0917 18:01:36.022628    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e1a3d22c-9336-478b-9229-0a4a69c76d9c-proxy-tls podName:e1a3d22c-9336-478b-9229-0a4a69c76d9c nodeName:}" failed. No retries permitted until 2026-09-17 18:01:36.522611338 +0000 UTC m=+1237.546216452 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/e1a3d22c-9336-478b-9229-0a4a69c76d9c-proxy-tls") pod "ensemble-graph-0dcf6-84fd5d8669-w4s5w" (UID: "e1a3d22c-9336-478b-9229-0a4a69c76d9c") : secret "ensemble-graph-0dcf6-serving-cert" not found
Sep 17 18:01:36.023050 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:01:36.023033    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e1a3d22c-9336-478b-9229-0a4a69c76d9c-openshift-service-ca-bundle\") pod \"ensemble-graph-0dcf6-84fd5d8669-w4s5w\" (UID: \"e1a3d22c-9336-478b-9229-0a4a69c76d9c\") " pod="kserve-ci-e2e-test/ensemble-graph-0dcf6-84fd5d8669-w4s5w"
Sep 17 18:01:36.149068 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:01:36.148983    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 18:01:36.527003 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:01:36.526925    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e1a3d22c-9336-478b-9229-0a4a69c76d9c-proxy-tls\") pod \"ensemble-graph-0dcf6-84fd5d8669-w4s5w\" (UID: \"e1a3d22c-9336-478b-9229-0a4a69c76d9c\") " pod="kserve-ci-e2e-test/ensemble-graph-0dcf6-84fd5d8669-w4s5w"
Sep 17 18:01:36.529266 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:01:36.529243    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e1a3d22c-9336-478b-9229-0a4a69c76d9c-proxy-tls\") pod \"ensemble-graph-0dcf6-84fd5d8669-w4s5w\" (UID: \"e1a3d22c-9336-478b-9229-0a4a69c76d9c\") " pod="kserve-ci-e2e-test/ensemble-graph-0dcf6-84fd5d8669-w4s5w"
Sep 17 18:01:36.718470 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:01:36.718426    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-0dcf6-84fd5d8669-w4s5w"
Sep 17 18:01:36.993991 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:01:36.993964    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-0dcf6-84fd5d8669-w4s5w"]
Sep 17 18:01:36.996013 ip-10-0-139-237 kubenswrapper[2567]: W0917 18:01:36.995987    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1a3d22c_9336_478b_9229_0a4a69c76d9c.slice/crio-9b3c5c2ad0c866f3ddec5772d550f1687b13edc85a1c133b13ff1ebd7da51701 WatchSource:0}: Error finding container 9b3c5c2ad0c866f3ddec5772d550f1687b13edc85a1c133b13ff1ebd7da51701: Status 404 returned error can't find the container with id 9b3c5c2ad0c866f3ddec5772d550f1687b13edc85a1c133b13ff1ebd7da51701
Sep 17 18:01:37.511216 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:01:37.511177    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-0dcf6-84fd5d8669-w4s5w" event={"ID":"e1a3d22c-9336-478b-9229-0a4a69c76d9c","Type":"ContainerStarted","Data":"f35200f2199cc6f4e21abacb69260d136bde60a4eb73f1f5676c56345ecf3c6d"}
Sep 17 18:01:37.511413 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:01:37.511236    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-0dcf6-84fd5d8669-w4s5w" event={"ID":"e1a3d22c-9336-478b-9229-0a4a69c76d9c","Type":"ContainerStarted","Data":"9b3c5c2ad0c866f3ddec5772d550f1687b13edc85a1c133b13ff1ebd7da51701"}
Sep 17 18:01:37.511413 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:01:37.511273    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/ensemble-graph-0dcf6-84fd5d8669-w4s5w"
Sep 17 18:01:37.538216 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:01:37.538173    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/ensemble-graph-0dcf6-84fd5d8669-w4s5w" podStartSLOduration=2.538158628 podStartE2EDuration="2.538158628s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:01:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:01:37.536140119 +0000 UTC m=+1238.559745256" watchObservedRunningTime="2026-09-17 18:01:37.538158628 +0000 UTC m=+1238.561763764"
Sep 17 18:01:40.509731 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:01:40.509688    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-081d8-66964d46f7-ghmwg" podUID="976221a4-d467-49c4-9b8b-174b8fba34e3" containerName="sequence-graph-081d8" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 18:01:40.510123 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:01:40.509790    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-081d8-66964d46f7-ghmwg"
Sep 17 18:01:43.475616 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:01:43.475518    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-8649f-predictor-f69888d4c-dvxpd" podUID="74c1ac49-8a02-45ec-b881-59bbb009d858" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Sep 17 18:01:43.521443 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:01:43.521417    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/ensemble-graph-0dcf6-84fd5d8669-w4s5w"
Sep 17 18:01:45.511400 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:01:45.511365    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-081d8-66964d46f7-ghmwg" podUID="976221a4-d467-49c4-9b8b-174b8fba34e3" containerName="sequence-graph-081d8" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 18:01:45.926549 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:01:45.926490    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-0dcf6-84fd5d8669-w4s5w"]
Sep 17 18:01:45.926791 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:01:45.926764    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/ensemble-graph-0dcf6-84fd5d8669-w4s5w" podUID="e1a3d22c-9336-478b-9229-0a4a69c76d9c" containerName="ensemble-graph-0dcf6" containerID="cri-o://f35200f2199cc6f4e21abacb69260d136bde60a4eb73f1f5676c56345ecf3c6d" gracePeriod=30
Sep 17 18:01:45.950643 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:01:45.950617    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 18:01:46.048690 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:01:46.048661    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 18:01:46.054196 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:01:46.054166    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-0dcf6-predictor-5d98597747-kxs58"]
Sep 17 18:01:46.054549 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:01:46.054499    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-0dcf6-predictor-5d98597747-kxs58" podUID="43e1e374-0ac1-4fa7-ab65-95ed213ba4ba" containerName="kube-rbac-proxy" containerID="cri-o://e508bf3da470d99aecece89b6623d082eb856945fcae0b51d54cecfefbc86914" gracePeriod=30
Sep 17 18:01:46.054701 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:01:46.054509    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-0dcf6-predictor-5d98597747-kxs58" podUID="43e1e374-0ac1-4fa7-ab65-95ed213ba4ba" containerName="kserve-container" containerID="cri-o://8607b90d2dd8f7fd887a7d95bd42d7abba4e204f4ca994a6fd91341865aa2d6c" gracePeriod=30
Sep 17 18:01:46.123695 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:01:46.123665    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 18:01:46.218499 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:01:46.218428    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 18:01:46.257499 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:01:46.257469    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-9d702-predictor-5967c4d597-nkqp8"]
Sep 17 18:01:46.259955 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:01:46.259934    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-9d702-predictor-5967c4d597-nkqp8"
Sep 17 18:01:46.262391 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:01:46.262364    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-9d702-predictor-serving-cert\""
Sep 17 18:01:46.262569 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:01:46.262551    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-9d702-kube-rbac-proxy-sar-config\""
Sep 17 18:01:46.271581 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:01:46.271561    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-9d702-predictor-5967c4d597-nkqp8"]
Sep 17 18:01:46.307335 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:01:46.307305    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 18:01:46.311492 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:01:46.311472    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x6fv8\" (UniqueName: \"kubernetes.io/projected/f161c155-a751-4207-a686-6154fb8949a9-kube-api-access-x6fv8\") pod \"error-404-isvc-9d702-predictor-5967c4d597-nkqp8\" (UID: \"f161c155-a751-4207-a686-6154fb8949a9\") " pod="kserve-ci-e2e-test/error-404-isvc-9d702-predictor-5967c4d597-nkqp8"
Sep 17 18:01:46.311582 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:01:46.311522    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f161c155-a751-4207-a686-6154fb8949a9-proxy-tls\") pod \"error-404-isvc-9d702-predictor-5967c4d597-nkqp8\" (UID: \"f161c155-a751-4207-a686-6154fb8949a9\") " pod="kserve-ci-e2e-test/error-404-isvc-9d702-predictor-5967c4d597-nkqp8"
Sep 17 18:01:46.311620 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:01:46.311588    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f161c155-a751-4207-a686-6154fb8949a9-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-9d702-predictor-5967c4d597-nkqp8\" (UID: \"f161c155-a751-4207-a686-6154fb8949a9\") " pod="kserve-ci-e2e-test/error-404-isvc-9d702-predictor-5967c4d597-nkqp8"
Sep 17 18:01:46.412132 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:01:46.412087    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f161c155-a751-4207-a686-6154fb8949a9-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-9d702-predictor-5967c4d597-nkqp8\" (UID: \"f161c155-a751-4207-a686-6154fb8949a9\") " pod="kserve-ci-e2e-test/error-404-isvc-9d702-predictor-5967c4d597-nkqp8"
Sep 17 18:01:46.412326 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:01:46.412140    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x6fv8\" (UniqueName: \"kubernetes.io/projected/f161c155-a751-4207-a686-6154fb8949a9-kube-api-access-x6fv8\") pod \"error-404-isvc-9d702-predictor-5967c4d597-nkqp8\" (UID: \"f161c155-a751-4207-a686-6154fb8949a9\") " pod="kserve-ci-e2e-test/error-404-isvc-9d702-predictor-5967c4d597-nkqp8"
Sep 17 18:01:46.412326 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:01:46.412185    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f161c155-a751-4207-a686-6154fb8949a9-proxy-tls\") pod \"error-404-isvc-9d702-predictor-5967c4d597-nkqp8\" (UID: \"f161c155-a751-4207-a686-6154fb8949a9\") " pod="kserve-ci-e2e-test/error-404-isvc-9d702-predictor-5967c4d597-nkqp8"
Sep 17 18:01:46.412326 ip-10-0-139-237 kubenswrapper[2567]: E0917 18:01:46.412289    2567 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-9d702-predictor-serving-cert: secret "error-404-isvc-9d702-predictor-serving-cert" not found
Sep 17 18:01:46.412454 ip-10-0-139-237 kubenswrapper[2567]: E0917 18:01:46.412351    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f161c155-a751-4207-a686-6154fb8949a9-proxy-tls podName:f161c155-a751-4207-a686-6154fb8949a9 nodeName:}" failed. No retries permitted until 2026-09-17 18:01:46.912330817 +0000 UTC m=+1247.935935945 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/f161c155-a751-4207-a686-6154fb8949a9-proxy-tls") pod "error-404-isvc-9d702-predictor-5967c4d597-nkqp8" (UID: "f161c155-a751-4207-a686-6154fb8949a9") : secret "error-404-isvc-9d702-predictor-serving-cert" not found
Sep 17 18:01:46.412840 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:01:46.412821    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f161c155-a751-4207-a686-6154fb8949a9-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-9d702-predictor-5967c4d597-nkqp8\" (UID: \"f161c155-a751-4207-a686-6154fb8949a9\") " pod="kserve-ci-e2e-test/error-404-isvc-9d702-predictor-5967c4d597-nkqp8"
Sep 17 18:01:46.421538 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:01:46.421511    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x6fv8\" (UniqueName: \"kubernetes.io/projected/f161c155-a751-4207-a686-6154fb8949a9-kube-api-access-x6fv8\") pod \"error-404-isvc-9d702-predictor-5967c4d597-nkqp8\" (UID: \"f161c155-a751-4207-a686-6154fb8949a9\") " pod="kserve-ci-e2e-test/error-404-isvc-9d702-predictor-5967c4d597-nkqp8"
Sep 17 18:01:46.549293 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:01:46.549258    2567 generic.go:356] "Generic (PLEG): container finished" podID="43e1e374-0ac1-4fa7-ab65-95ed213ba4ba" containerID="e508bf3da470d99aecece89b6623d082eb856945fcae0b51d54cecfefbc86914" exitCode=2
Sep 17 18:01:46.549660 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:01:46.549326    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-0dcf6-predictor-5d98597747-kxs58" event={"ID":"43e1e374-0ac1-4fa7-ab65-95ed213ba4ba","Type":"ContainerDied","Data":"e508bf3da470d99aecece89b6623d082eb856945fcae0b51d54cecfefbc86914"}
Sep 17 18:01:46.584694 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:01:46.584659    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 18:01:46.906624 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:01:46.906521    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 18:01:46.916647 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:01:46.916623    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f161c155-a751-4207-a686-6154fb8949a9-proxy-tls\") pod \"error-404-isvc-9d702-predictor-5967c4d597-nkqp8\" (UID: \"f161c155-a751-4207-a686-6154fb8949a9\") " pod="kserve-ci-e2e-test/error-404-isvc-9d702-predictor-5967c4d597-nkqp8"
Sep 17 18:01:46.918979 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:01:46.918958    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f161c155-a751-4207-a686-6154fb8949a9-proxy-tls\") pod \"error-404-isvc-9d702-predictor-5967c4d597-nkqp8\" (UID: \"f161c155-a751-4207-a686-6154fb8949a9\") " pod="kserve-ci-e2e-test/error-404-isvc-9d702-predictor-5967c4d597-nkqp8"
Sep 17 18:01:47.170287 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:01:47.170197    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-9d702-predictor-5967c4d597-nkqp8"
Sep 17 18:01:47.441908 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:01:47.441881    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-9d702-predictor-5967c4d597-nkqp8"]
Sep 17 18:01:47.444149 ip-10-0-139-237 kubenswrapper[2567]: W0917 18:01:47.444117    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf161c155_a751_4207_a686_6154fb8949a9.slice/crio-6b44999d3af0a12ef9f678b909ead42cda1afb6f1aadd885312765da5be02192 WatchSource:0}: Error finding container 6b44999d3af0a12ef9f678b909ead42cda1afb6f1aadd885312765da5be02192: Status 404 returned error can't find the container with id 6b44999d3af0a12ef9f678b909ead42cda1afb6f1aadd885312765da5be02192
Sep 17 18:01:47.555782 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:01:47.555749    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-9d702-predictor-5967c4d597-nkqp8" event={"ID":"f161c155-a751-4207-a686-6154fb8949a9","Type":"ContainerStarted","Data":"e63913250ccc577d9238c3f4d7ad26b296c597759a6968a164fec61a9f3bb50e"}
Sep 17 18:01:47.556058 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:01:47.555826    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-9d702-predictor-5967c4d597-nkqp8" event={"ID":"f161c155-a751-4207-a686-6154fb8949a9","Type":"ContainerStarted","Data":"6b44999d3af0a12ef9f678b909ead42cda1afb6f1aadd885312765da5be02192"}
Sep 17 18:01:48.519394 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:01:48.519358    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-0dcf6-84fd5d8669-w4s5w" podUID="e1a3d22c-9336-478b-9229-0a4a69c76d9c" containerName="ensemble-graph-0dcf6" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 18:01:48.563214 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:01:48.563183    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-9d702-predictor-5967c4d597-nkqp8" event={"ID":"f161c155-a751-4207-a686-6154fb8949a9","Type":"ContainerStarted","Data":"e91634bff0c24472ab73401dd9d8909baf410768c552e1665de1269316314a01"}
Sep 17 18:01:48.563595 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:01:48.563275    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-9d702-predictor-5967c4d597-nkqp8"
Sep 17 18:01:48.582439 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:01:48.582393    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-9d702-predictor-5967c4d597-nkqp8" podStartSLOduration=2.582379807 podStartE2EDuration="2.582379807s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:01:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:01:48.580634004 +0000 UTC m=+1249.604239161" watchObservedRunningTime="2026-09-17 18:01:48.582379807 +0000 UTC m=+1249.605984943"
Sep 17 18:01:49.300805 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:01:49.300784    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-0dcf6-predictor-5d98597747-kxs58"
Sep 17 18:01:49.438134 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:01:49.437953    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/43e1e374-0ac1-4fa7-ab65-95ed213ba4ba-proxy-tls\") pod \"43e1e374-0ac1-4fa7-ab65-95ed213ba4ba\" (UID: \"43e1e374-0ac1-4fa7-ab65-95ed213ba4ba\") "
Sep 17 18:01:49.438134 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:01:49.438066    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/43e1e374-0ac1-4fa7-ab65-95ed213ba4ba-kube-rbac-proxy-sar-config\") pod \"43e1e374-0ac1-4fa7-ab65-95ed213ba4ba\" (UID: \"43e1e374-0ac1-4fa7-ab65-95ed213ba4ba\") "
Sep 17 18:01:49.438134 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:01:49.438102    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4nr5r\" (UniqueName: \"kubernetes.io/projected/43e1e374-0ac1-4fa7-ab65-95ed213ba4ba-kube-api-access-4nr5r\") pod \"43e1e374-0ac1-4fa7-ab65-95ed213ba4ba\" (UID: \"43e1e374-0ac1-4fa7-ab65-95ed213ba4ba\") "
Sep 17 18:01:49.438573 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:01:49.438522    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43e1e374-0ac1-4fa7-ab65-95ed213ba4ba-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "43e1e374-0ac1-4fa7-ab65-95ed213ba4ba" (UID: "43e1e374-0ac1-4fa7-ab65-95ed213ba4ba"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:01:49.440081 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:01:49.440047    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/43e1e374-0ac1-4fa7-ab65-95ed213ba4ba-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "43e1e374-0ac1-4fa7-ab65-95ed213ba4ba" (UID: "43e1e374-0ac1-4fa7-ab65-95ed213ba4ba"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:01:49.440177 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:01:49.440131    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43e1e374-0ac1-4fa7-ab65-95ed213ba4ba-kube-api-access-4nr5r" (OuterVolumeSpecName: "kube-api-access-4nr5r") pod "43e1e374-0ac1-4fa7-ab65-95ed213ba4ba" (UID: "43e1e374-0ac1-4fa7-ab65-95ed213ba4ba"). InnerVolumeSpecName "kube-api-access-4nr5r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:01:49.539347 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:01:49.539320    2567 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/43e1e374-0ac1-4fa7-ab65-95ed213ba4ba-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-237.ec2.internal\" DevicePath \"\""
Sep 17 18:01:49.539347 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:01:49.539344    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4nr5r\" (UniqueName: \"kubernetes.io/projected/43e1e374-0ac1-4fa7-ab65-95ed213ba4ba-kube-api-access-4nr5r\") on node \"ip-10-0-139-237.ec2.internal\" DevicePath \"\""
Sep 17 18:01:49.539502 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:01:49.539357    2567 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/43e1e374-0ac1-4fa7-ab65-95ed213ba4ba-proxy-tls\") on node \"ip-10-0-139-237.ec2.internal\" DevicePath \"\""
Sep 17 18:01:49.568896 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:01:49.568869    2567 generic.go:356] "Generic (PLEG): container finished" podID="43e1e374-0ac1-4fa7-ab65-95ed213ba4ba" containerID="8607b90d2dd8f7fd887a7d95bd42d7abba4e204f4ca994a6fd91341865aa2d6c" exitCode=0
Sep 17 18:01:49.569247 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:01:49.568918    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-0dcf6-predictor-5d98597747-kxs58"
Sep 17 18:01:49.569247 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:01:49.568919    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-0dcf6-predictor-5d98597747-kxs58" event={"ID":"43e1e374-0ac1-4fa7-ab65-95ed213ba4ba","Type":"ContainerDied","Data":"8607b90d2dd8f7fd887a7d95bd42d7abba4e204f4ca994a6fd91341865aa2d6c"}
Sep 17 18:01:49.569247 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:01:49.568980    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-0dcf6-predictor-5d98597747-kxs58" event={"ID":"43e1e374-0ac1-4fa7-ab65-95ed213ba4ba","Type":"ContainerDied","Data":"a1902d5a69c8bb887331df0789828cbcc8d9491226f372aff53a5cb8e9f4acb2"}
Sep 17 18:01:49.569247 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:01:49.569008    2567 scope.go:117] "RemoveContainer" containerID="e508bf3da470d99aecece89b6623d082eb856945fcae0b51d54cecfefbc86914"
Sep 17 18:01:49.569461 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:01:49.569355    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-9d702-predictor-5967c4d597-nkqp8"
Sep 17 18:01:49.570623 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:01:49.570596    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-9d702-predictor-5967c4d597-nkqp8" podUID="f161c155-a751-4207-a686-6154fb8949a9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Sep 17 18:01:49.580118 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:01:49.580044    2567 scope.go:117] "RemoveContainer" containerID="8607b90d2dd8f7fd887a7d95bd42d7abba4e204f4ca994a6fd91341865aa2d6c"
Sep 17 18:01:49.587656 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:01:49.587641    2567 scope.go:117] "RemoveContainer" containerID="e508bf3da470d99aecece89b6623d082eb856945fcae0b51d54cecfefbc86914"
Sep 17 18:01:49.587912 ip-10-0-139-237 kubenswrapper[2567]: E0917 18:01:49.587894    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e508bf3da470d99aecece89b6623d082eb856945fcae0b51d54cecfefbc86914\": container with ID starting with e508bf3da470d99aecece89b6623d082eb856945fcae0b51d54cecfefbc86914 not found: ID does not exist" containerID="e508bf3da470d99aecece89b6623d082eb856945fcae0b51d54cecfefbc86914"
Sep 17 18:01:49.587969 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:01:49.587921    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e508bf3da470d99aecece89b6623d082eb856945fcae0b51d54cecfefbc86914"} err="failed to get container status \"e508bf3da470d99aecece89b6623d082eb856945fcae0b51d54cecfefbc86914\": rpc error: code = NotFound desc = could not find container \"e508bf3da470d99aecece89b6623d082eb856945fcae0b51d54cecfefbc86914\": container with ID starting with e508bf3da470d99aecece89b6623d082eb856945fcae0b51d54cecfefbc86914 not found: ID does not exist"
Sep 17 18:01:49.587969 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:01:49.587935    2567 scope.go:117] "RemoveContainer" containerID="8607b90d2dd8f7fd887a7d95bd42d7abba4e204f4ca994a6fd91341865aa2d6c"
Sep 17 18:01:49.588173 ip-10-0-139-237 kubenswrapper[2567]: E0917 18:01:49.588155    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8607b90d2dd8f7fd887a7d95bd42d7abba4e204f4ca994a6fd91341865aa2d6c\": container with ID starting with 8607b90d2dd8f7fd887a7d95bd42d7abba4e204f4ca994a6fd91341865aa2d6c not found: ID does not exist" containerID="8607b90d2dd8f7fd887a7d95bd42d7abba4e204f4ca994a6fd91341865aa2d6c"
Sep 17 18:01:49.588207 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:01:49.588180    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8607b90d2dd8f7fd887a7d95bd42d7abba4e204f4ca994a6fd91341865aa2d6c"} err="failed to get container status \"8607b90d2dd8f7fd887a7d95bd42d7abba4e204f4ca994a6fd91341865aa2d6c\": rpc error: code = NotFound desc = could not find container \"8607b90d2dd8f7fd887a7d95bd42d7abba4e204f4ca994a6fd91341865aa2d6c\": container with ID starting with 8607b90d2dd8f7fd887a7d95bd42d7abba4e204f4ca994a6fd91341865aa2d6c not found: ID does not exist"
Sep 17 18:01:49.603879 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:01:49.603859    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-0dcf6-predictor-5d98597747-kxs58"]
Sep 17 18:01:49.615579 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:01:49.614275    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-0dcf6-predictor-5d98597747-kxs58"]
Sep 17 18:01:50.271415 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:01:50.271364    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-0dcf6-predictor-5d98597747-kxs58" podUID="43e1e374-0ac1-4fa7-ab65-95ed213ba4ba" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.32:8643/healthz\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Sep 17 18:01:50.509320 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:01:50.509286    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-081d8-66964d46f7-ghmwg" podUID="976221a4-d467-49c4-9b8b-174b8fba34e3" containerName="sequence-graph-081d8" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 18:01:50.574106 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:01:50.574070    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-9d702-predictor-5967c4d597-nkqp8" podUID="f161c155-a751-4207-a686-6154fb8949a9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Sep 17 18:01:51.287440 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:01:51.287406    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="43e1e374-0ac1-4fa7-ab65-95ed213ba4ba" path="/var/lib/kubelet/pods/43e1e374-0ac1-4fa7-ab65-95ed213ba4ba/volumes"
Sep 17 18:01:53.475795 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:01:53.475755    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-8649f-predictor-f69888d4c-dvxpd" podUID="74c1ac49-8a02-45ec-b881-59bbb009d858" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Sep 17 18:01:53.518743 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:01:53.518710    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-0dcf6-84fd5d8669-w4s5w" podUID="e1a3d22c-9336-478b-9229-0a4a69c76d9c" containerName="ensemble-graph-0dcf6" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 18:01:55.508949 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:01:55.508911    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-081d8-66964d46f7-ghmwg" podUID="976221a4-d467-49c4-9b8b-174b8fba34e3" containerName="sequence-graph-081d8" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 18:01:55.578125 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:01:55.578099    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-9d702-predictor-5967c4d597-nkqp8"
Sep 17 18:01:55.578545 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:01:55.578504    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-9d702-predictor-5967c4d597-nkqp8" podUID="f161c155-a751-4207-a686-6154fb8949a9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Sep 17 18:01:55.933637 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:01:55.933612    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-081d8-66964d46f7-ghmwg"
Sep 17 18:01:55.995955 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:01:55.995930    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/976221a4-d467-49c4-9b8b-174b8fba34e3-openshift-service-ca-bundle\") pod \"976221a4-d467-49c4-9b8b-174b8fba34e3\" (UID: \"976221a4-d467-49c4-9b8b-174b8fba34e3\") "
Sep 17 18:01:55.996091 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:01:55.996019    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/976221a4-d467-49c4-9b8b-174b8fba34e3-proxy-tls\") pod \"976221a4-d467-49c4-9b8b-174b8fba34e3\" (UID: \"976221a4-d467-49c4-9b8b-174b8fba34e3\") "
Sep 17 18:01:55.996329 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:01:55.996303    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/976221a4-d467-49c4-9b8b-174b8fba34e3-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "976221a4-d467-49c4-9b8b-174b8fba34e3" (UID: "976221a4-d467-49c4-9b8b-174b8fba34e3"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:01:55.997960 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:01:55.997942    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/976221a4-d467-49c4-9b8b-174b8fba34e3-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "976221a4-d467-49c4-9b8b-174b8fba34e3" (UID: "976221a4-d467-49c4-9b8b-174b8fba34e3"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:01:56.096846 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:01:56.096776    2567 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/976221a4-d467-49c4-9b8b-174b8fba34e3-openshift-service-ca-bundle\") on node \"ip-10-0-139-237.ec2.internal\" DevicePath \"\""
Sep 17 18:01:56.096846 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:01:56.096800    2567 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/976221a4-d467-49c4-9b8b-174b8fba34e3-proxy-tls\") on node \"ip-10-0-139-237.ec2.internal\" DevicePath \"\""
Sep 17 18:01:56.601667 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:01:56.601638    2567 generic.go:356] "Generic (PLEG): container finished" podID="976221a4-d467-49c4-9b8b-174b8fba34e3" containerID="04532fda5677a9ae9e4e1c86e61fc95d13e3d2bddc659911114eccd5b42d056d" exitCode=0
Sep 17 18:01:56.602062 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:01:56.601683    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-081d8-66964d46f7-ghmwg"
Sep 17 18:01:56.602062 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:01:56.601721    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-081d8-66964d46f7-ghmwg" event={"ID":"976221a4-d467-49c4-9b8b-174b8fba34e3","Type":"ContainerDied","Data":"04532fda5677a9ae9e4e1c86e61fc95d13e3d2bddc659911114eccd5b42d056d"}
Sep 17 18:01:56.602062 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:01:56.601774    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-081d8-66964d46f7-ghmwg" event={"ID":"976221a4-d467-49c4-9b8b-174b8fba34e3","Type":"ContainerDied","Data":"d2d4b7e8d53870bffdb563df5ff76924c059d1a947d7355430bfd50c2a5aa8d6"}
Sep 17 18:01:56.602062 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:01:56.601803    2567 scope.go:117] "RemoveContainer" containerID="04532fda5677a9ae9e4e1c86e61fc95d13e3d2bddc659911114eccd5b42d056d"
Sep 17 18:01:56.610620 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:01:56.610602    2567 scope.go:117] "RemoveContainer" containerID="04532fda5677a9ae9e4e1c86e61fc95d13e3d2bddc659911114eccd5b42d056d"
Sep 17 18:01:56.610899 ip-10-0-139-237 kubenswrapper[2567]: E0917 18:01:56.610873    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"04532fda5677a9ae9e4e1c86e61fc95d13e3d2bddc659911114eccd5b42d056d\": container with ID starting with 04532fda5677a9ae9e4e1c86e61fc95d13e3d2bddc659911114eccd5b42d056d not found: ID does not exist" containerID="04532fda5677a9ae9e4e1c86e61fc95d13e3d2bddc659911114eccd5b42d056d"
Sep 17 18:01:56.610955 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:01:56.610911    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04532fda5677a9ae9e4e1c86e61fc95d13e3d2bddc659911114eccd5b42d056d"} err="failed to get container status \"04532fda5677a9ae9e4e1c86e61fc95d13e3d2bddc659911114eccd5b42d056d\": rpc error: code = NotFound desc = could not find container \"04532fda5677a9ae9e4e1c86e61fc95d13e3d2bddc659911114eccd5b42d056d\": container with ID starting with 04532fda5677a9ae9e4e1c86e61fc95d13e3d2bddc659911114eccd5b42d056d not found: ID does not exist"
Sep 17 18:01:56.624958 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:01:56.624924    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-081d8-66964d46f7-ghmwg"]
Sep 17 18:01:56.629448 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:01:56.629426    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-081d8-66964d46f7-ghmwg"]
Sep 17 18:01:57.286382 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:01:57.286351    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="976221a4-d467-49c4-9b8b-174b8fba34e3" path="/var/lib/kubelet/pods/976221a4-d467-49c4-9b8b-174b8fba34e3/volumes"
Sep 17 18:01:58.519721 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:01:58.519682    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-0dcf6-84fd5d8669-w4s5w" podUID="e1a3d22c-9336-478b-9229-0a4a69c76d9c" containerName="ensemble-graph-0dcf6" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 18:01:58.520157 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:01:58.519777    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/ensemble-graph-0dcf6-84fd5d8669-w4s5w"
Sep 17 18:02:03.475938 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:02:03.475902    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-8649f-predictor-f69888d4c-dvxpd" podUID="74c1ac49-8a02-45ec-b881-59bbb009d858" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Sep 17 18:02:03.519509 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:02:03.519474    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-0dcf6-84fd5d8669-w4s5w" podUID="e1a3d22c-9336-478b-9229-0a4a69c76d9c" containerName="ensemble-graph-0dcf6" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 18:02:05.578909 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:02:05.578869    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-9d702-predictor-5967c4d597-nkqp8" podUID="f161c155-a751-4207-a686-6154fb8949a9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Sep 17 18:02:08.519116 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:02:08.519073    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-0dcf6-84fd5d8669-w4s5w" podUID="e1a3d22c-9336-478b-9229-0a4a69c76d9c" containerName="ensemble-graph-0dcf6" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 18:02:13.476544 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:02:13.476498    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-8649f-predictor-f69888d4c-dvxpd"
Sep 17 18:02:13.519923 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:02:13.519888    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-0dcf6-84fd5d8669-w4s5w" podUID="e1a3d22c-9336-478b-9229-0a4a69c76d9c" containerName="ensemble-graph-0dcf6" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 18:02:15.578754 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:02:15.578713    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-9d702-predictor-5967c4d597-nkqp8" podUID="f161c155-a751-4207-a686-6154fb8949a9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Sep 17 18:02:16.102052 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:02:16.102030    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-0dcf6-84fd5d8669-w4s5w"
Sep 17 18:02:16.267939 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:02:16.267862    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e1a3d22c-9336-478b-9229-0a4a69c76d9c-openshift-service-ca-bundle\") pod \"e1a3d22c-9336-478b-9229-0a4a69c76d9c\" (UID: \"e1a3d22c-9336-478b-9229-0a4a69c76d9c\") "
Sep 17 18:02:16.267939 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:02:16.267909    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e1a3d22c-9336-478b-9229-0a4a69c76d9c-proxy-tls\") pod \"e1a3d22c-9336-478b-9229-0a4a69c76d9c\" (UID: \"e1a3d22c-9336-478b-9229-0a4a69c76d9c\") "
Sep 17 18:02:16.268210 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:02:16.268187    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e1a3d22c-9336-478b-9229-0a4a69c76d9c-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "e1a3d22c-9336-478b-9229-0a4a69c76d9c" (UID: "e1a3d22c-9336-478b-9229-0a4a69c76d9c"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:02:16.270072 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:02:16.270043    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e1a3d22c-9336-478b-9229-0a4a69c76d9c-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "e1a3d22c-9336-478b-9229-0a4a69c76d9c" (UID: "e1a3d22c-9336-478b-9229-0a4a69c76d9c"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:02:16.368960 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:02:16.368932    2567 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e1a3d22c-9336-478b-9229-0a4a69c76d9c-openshift-service-ca-bundle\") on node \"ip-10-0-139-237.ec2.internal\" DevicePath \"\""
Sep 17 18:02:16.368960 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:02:16.368958    2567 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e1a3d22c-9336-478b-9229-0a4a69c76d9c-proxy-tls\") on node \"ip-10-0-139-237.ec2.internal\" DevicePath \"\""
Sep 17 18:02:16.684294 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:02:16.684257    2567 generic.go:356] "Generic (PLEG): container finished" podID="e1a3d22c-9336-478b-9229-0a4a69c76d9c" containerID="f35200f2199cc6f4e21abacb69260d136bde60a4eb73f1f5676c56345ecf3c6d" exitCode=0
Sep 17 18:02:16.684716 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:02:16.684309    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-0dcf6-84fd5d8669-w4s5w"
Sep 17 18:02:16.684716 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:02:16.684349    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-0dcf6-84fd5d8669-w4s5w" event={"ID":"e1a3d22c-9336-478b-9229-0a4a69c76d9c","Type":"ContainerDied","Data":"f35200f2199cc6f4e21abacb69260d136bde60a4eb73f1f5676c56345ecf3c6d"}
Sep 17 18:02:16.684716 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:02:16.684402    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-0dcf6-84fd5d8669-w4s5w" event={"ID":"e1a3d22c-9336-478b-9229-0a4a69c76d9c","Type":"ContainerDied","Data":"9b3c5c2ad0c866f3ddec5772d550f1687b13edc85a1c133b13ff1ebd7da51701"}
Sep 17 18:02:16.684716 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:02:16.684437    2567 scope.go:117] "RemoveContainer" containerID="f35200f2199cc6f4e21abacb69260d136bde60a4eb73f1f5676c56345ecf3c6d"
Sep 17 18:02:16.693296 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:02:16.693278    2567 scope.go:117] "RemoveContainer" containerID="f35200f2199cc6f4e21abacb69260d136bde60a4eb73f1f5676c56345ecf3c6d"
Sep 17 18:02:16.693549 ip-10-0-139-237 kubenswrapper[2567]: E0917 18:02:16.693512    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f35200f2199cc6f4e21abacb69260d136bde60a4eb73f1f5676c56345ecf3c6d\": container with ID starting with f35200f2199cc6f4e21abacb69260d136bde60a4eb73f1f5676c56345ecf3c6d not found: ID does not exist" containerID="f35200f2199cc6f4e21abacb69260d136bde60a4eb73f1f5676c56345ecf3c6d"
Sep 17 18:02:16.693633 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:02:16.693564    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f35200f2199cc6f4e21abacb69260d136bde60a4eb73f1f5676c56345ecf3c6d"} err="failed to get container status \"f35200f2199cc6f4e21abacb69260d136bde60a4eb73f1f5676c56345ecf3c6d\": rpc error: code = NotFound desc = could not find container \"f35200f2199cc6f4e21abacb69260d136bde60a4eb73f1f5676c56345ecf3c6d\": container with ID starting with f35200f2199cc6f4e21abacb69260d136bde60a4eb73f1f5676c56345ecf3c6d not found: ID does not exist"
Sep 17 18:02:16.711957 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:02:16.711924    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-0dcf6-84fd5d8669-w4s5w"]
Sep 17 18:02:16.715562 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:02:16.715540    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-0dcf6-84fd5d8669-w4s5w"]
Sep 17 18:02:17.286108 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:02:17.286067    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e1a3d22c-9336-478b-9229-0a4a69c76d9c" path="/var/lib/kubelet/pods/e1a3d22c-9336-478b-9229-0a4a69c76d9c/volumes"
Sep 17 18:02:18.915212 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:02:18.915174    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 18:02:20.036422 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:02:20.036386    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 18:02:25.579224 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:02:25.579186    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-9d702-predictor-5967c4d597-nkqp8" podUID="f161c155-a751-4207-a686-6154fb8949a9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Sep 17 18:02:26.085492 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:02:26.085456    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 18:02:26.087739 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:02:26.087714    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/sequence-graph-8649f-fc945575-t7558"]
Sep 17 18:02:26.088038 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:02:26.088026    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="976221a4-d467-49c4-9b8b-174b8fba34e3" containerName="sequence-graph-081d8"
Sep 17 18:02:26.088083 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:02:26.088039    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="976221a4-d467-49c4-9b8b-174b8fba34e3" containerName="sequence-graph-081d8"
Sep 17 18:02:26.088083 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:02:26.088049    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="43e1e374-0ac1-4fa7-ab65-95ed213ba4ba" containerName="kube-rbac-proxy"
Sep 17 18:02:26.088083 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:02:26.088054    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="43e1e374-0ac1-4fa7-ab65-95ed213ba4ba" containerName="kube-rbac-proxy"
Sep 17 18:02:26.088083 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:02:26.088068    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e1a3d22c-9336-478b-9229-0a4a69c76d9c" containerName="ensemble-graph-0dcf6"
Sep 17 18:02:26.088083 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:02:26.088074    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="e1a3d22c-9336-478b-9229-0a4a69c76d9c" containerName="ensemble-graph-0dcf6"
Sep 17 18:02:26.088083 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:02:26.088082    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="43e1e374-0ac1-4fa7-ab65-95ed213ba4ba" containerName="kserve-container"
Sep 17 18:02:26.088291 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:02:26.088088    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="43e1e374-0ac1-4fa7-ab65-95ed213ba4ba" containerName="kserve-container"
Sep 17 18:02:26.088291 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:02:26.088149    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="43e1e374-0ac1-4fa7-ab65-95ed213ba4ba" containerName="kserve-container"
Sep 17 18:02:26.088291 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:02:26.088158    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="43e1e374-0ac1-4fa7-ab65-95ed213ba4ba" containerName="kube-rbac-proxy"
Sep 17 18:02:26.088291 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:02:26.088164    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="e1a3d22c-9336-478b-9229-0a4a69c76d9c" containerName="ensemble-graph-0dcf6"
Sep 17 18:02:26.088291 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:02:26.088172    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="976221a4-d467-49c4-9b8b-174b8fba34e3" containerName="sequence-graph-081d8"
Sep 17 18:02:26.089912 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:02:26.089897    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-8649f-fc945575-t7558"
Sep 17 18:02:26.106009 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:02:26.105992    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-8649f-serving-cert\""
Sep 17 18:02:26.106478 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:02:26.106465    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-8649f-kube-rbac-proxy-sar-config\""
Sep 17 18:02:26.115273 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:02:26.115253    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-8649f-fc945575-t7558"]
Sep 17 18:02:26.140350 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:02:26.140328    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/34575ee3-56a0-46aa-9055-7f1dbd70b55f-openshift-service-ca-bundle\") pod \"sequence-graph-8649f-fc945575-t7558\" (UID: \"34575ee3-56a0-46aa-9055-7f1dbd70b55f\") " pod="kserve-ci-e2e-test/sequence-graph-8649f-fc945575-t7558"
Sep 17 18:02:26.140445 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:02:26.140355    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/34575ee3-56a0-46aa-9055-7f1dbd70b55f-proxy-tls\") pod \"sequence-graph-8649f-fc945575-t7558\" (UID: \"34575ee3-56a0-46aa-9055-7f1dbd70b55f\") " pod="kserve-ci-e2e-test/sequence-graph-8649f-fc945575-t7558"
Sep 17 18:02:26.241413 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:02:26.241380    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/34575ee3-56a0-46aa-9055-7f1dbd70b55f-openshift-service-ca-bundle\") pod \"sequence-graph-8649f-fc945575-t7558\" (UID: \"34575ee3-56a0-46aa-9055-7f1dbd70b55f\") " pod="kserve-ci-e2e-test/sequence-graph-8649f-fc945575-t7558"
Sep 17 18:02:26.241413 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:02:26.241417    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/34575ee3-56a0-46aa-9055-7f1dbd70b55f-proxy-tls\") pod \"sequence-graph-8649f-fc945575-t7558\" (UID: \"34575ee3-56a0-46aa-9055-7f1dbd70b55f\") " pod="kserve-ci-e2e-test/sequence-graph-8649f-fc945575-t7558"
Sep 17 18:02:26.241638 ip-10-0-139-237 kubenswrapper[2567]: E0917 18:02:26.241601    2567 secret.go:189] Couldn't get secret kserve-ci-e2e-test/sequence-graph-8649f-serving-cert: secret "sequence-graph-8649f-serving-cert" not found
Sep 17 18:02:26.241680 ip-10-0-139-237 kubenswrapper[2567]: E0917 18:02:26.241670    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/34575ee3-56a0-46aa-9055-7f1dbd70b55f-proxy-tls podName:34575ee3-56a0-46aa-9055-7f1dbd70b55f nodeName:}" failed. No retries permitted until 2026-09-17 18:02:26.741649561 +0000 UTC m=+1287.765254674 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/34575ee3-56a0-46aa-9055-7f1dbd70b55f-proxy-tls") pod "sequence-graph-8649f-fc945575-t7558" (UID: "34575ee3-56a0-46aa-9055-7f1dbd70b55f") : secret "sequence-graph-8649f-serving-cert" not found
Sep 17 18:02:26.242142 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:02:26.242123    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/34575ee3-56a0-46aa-9055-7f1dbd70b55f-openshift-service-ca-bundle\") pod \"sequence-graph-8649f-fc945575-t7558\" (UID: \"34575ee3-56a0-46aa-9055-7f1dbd70b55f\") " pod="kserve-ci-e2e-test/sequence-graph-8649f-fc945575-t7558"
Sep 17 18:02:26.411967 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:02:26.411882    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 18:02:26.746993 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:02:26.746897    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/34575ee3-56a0-46aa-9055-7f1dbd70b55f-proxy-tls\") pod \"sequence-graph-8649f-fc945575-t7558\" (UID: \"34575ee3-56a0-46aa-9055-7f1dbd70b55f\") " pod="kserve-ci-e2e-test/sequence-graph-8649f-fc945575-t7558"
Sep 17 18:02:26.749311 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:02:26.749289    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/34575ee3-56a0-46aa-9055-7f1dbd70b55f-proxy-tls\") pod \"sequence-graph-8649f-fc945575-t7558\" (UID: \"34575ee3-56a0-46aa-9055-7f1dbd70b55f\") " pod="kserve-ci-e2e-test/sequence-graph-8649f-fc945575-t7558"
Sep 17 18:02:26.999526 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:02:26.999434    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-8649f-fc945575-t7558"
Sep 17 18:02:27.267636 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:02:27.267611    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-8649f-fc945575-t7558"]
Sep 17 18:02:27.270240 ip-10-0-139-237 kubenswrapper[2567]: W0917 18:02:27.270212    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34575ee3_56a0_46aa_9055_7f1dbd70b55f.slice/crio-b0aa9014ae3de90d59fe665df15fc4fcde578f46839af3262c31f98988fc6dc7 WatchSource:0}: Error finding container b0aa9014ae3de90d59fe665df15fc4fcde578f46839af3262c31f98988fc6dc7: Status 404 returned error can't find the container with id b0aa9014ae3de90d59fe665df15fc4fcde578f46839af3262c31f98988fc6dc7
Sep 17 18:02:27.731360 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:02:27.731322    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-8649f-fc945575-t7558" event={"ID":"34575ee3-56a0-46aa-9055-7f1dbd70b55f","Type":"ContainerStarted","Data":"e9a18029f62ad844a5df64aa95d9abee3e9dae3c8f7ab9299ba7f38fe1e4fb00"}
Sep 17 18:02:27.731580 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:02:27.731371    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-8649f-fc945575-t7558" event={"ID":"34575ee3-56a0-46aa-9055-7f1dbd70b55f","Type":"ContainerStarted","Data":"b0aa9014ae3de90d59fe665df15fc4fcde578f46839af3262c31f98988fc6dc7"}
Sep 17 18:02:27.731580 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:02:27.731407    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-8649f-fc945575-t7558"
Sep 17 18:02:27.749047 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:02:27.749003    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/sequence-graph-8649f-fc945575-t7558" podStartSLOduration=1.7489907420000002 podStartE2EDuration="1.748990742s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:02:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:02:27.747614616 +0000 UTC m=+1288.771219777" watchObservedRunningTime="2026-09-17 18:02:27.748990742 +0000 UTC m=+1288.772595878"
Sep 17 18:02:33.741292 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:02:33.741262    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/sequence-graph-8649f-fc945575-t7558"
Sep 17 18:02:35.579674 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:02:35.579646    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-9d702-predictor-5967c4d597-nkqp8"
Sep 17 18:02:36.081810 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:02:36.081777    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-8649f-fc945575-t7558"]
Sep 17 18:02:36.082016 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:02:36.081987    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/sequence-graph-8649f-fc945575-t7558" podUID="34575ee3-56a0-46aa-9055-7f1dbd70b55f" containerName="sequence-graph-8649f" containerID="cri-o://e9a18029f62ad844a5df64aa95d9abee3e9dae3c8f7ab9299ba7f38fe1e4fb00" gracePeriod=30
Sep 17 18:02:36.110484 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:02:36.110452    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 18:02:36.214704 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:02:36.214670    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 18:02:36.227688 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:02:36.227657    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-8649f-predictor-f69888d4c-dvxpd"]
Sep 17 18:02:36.227937 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:02:36.227913    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-8649f-predictor-f69888d4c-dvxpd" podUID="74c1ac49-8a02-45ec-b881-59bbb009d858" containerName="kserve-container" containerID="cri-o://848fd758bfc502873a4860495a96c1f00150c691a8c6e9d8da7d66d0a0093f16" gracePeriod=30
Sep 17 18:02:36.228058 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:02:36.227952    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-8649f-predictor-f69888d4c-dvxpd" podUID="74c1ac49-8a02-45ec-b881-59bbb009d858" containerName="kube-rbac-proxy" containerID="cri-o://c6290751b1a7129c132628a82b06856ce94926f3149f6273c38b4d5cb8b039fa" gracePeriod=30
Sep 17 18:02:36.307480 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:02:36.307446    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 18:02:36.390797 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:02:36.390768    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 18:02:36.416960 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:02:36.416934    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-8eede-predictor-5d489c9599-98jkr"]
Sep 17 18:02:36.419666 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:02:36.419645    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-8eede-predictor-5d489c9599-98jkr"
Sep 17 18:02:36.422362 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:02:36.422339    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-8eede-kube-rbac-proxy-sar-config\""
Sep 17 18:02:36.422362 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:02:36.422359    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-8eede-predictor-serving-cert\""
Sep 17 18:02:36.432412 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:02:36.432392    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-8eede-predictor-5d489c9599-98jkr"]
Sep 17 18:02:36.457592 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:02:36.457565    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 18:02:36.524980 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:02:36.524955    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/73a12907-2862-4ed2-9a2b-ef500261948f-proxy-tls\") pod \"error-404-isvc-8eede-predictor-5d489c9599-98jkr\" (UID: \"73a12907-2862-4ed2-9a2b-ef500261948f\") " pod="kserve-ci-e2e-test/error-404-isvc-8eede-predictor-5d489c9599-98jkr"
Sep 17 18:02:36.525095 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:02:36.525000    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/73a12907-2862-4ed2-9a2b-ef500261948f-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-8eede-predictor-5d489c9599-98jkr\" (UID: \"73a12907-2862-4ed2-9a2b-ef500261948f\") " pod="kserve-ci-e2e-test/error-404-isvc-8eede-predictor-5d489c9599-98jkr"
Sep 17 18:02:36.525095 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:02:36.525029    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k2pwn\" (UniqueName: \"kubernetes.io/projected/73a12907-2862-4ed2-9a2b-ef500261948f-kube-api-access-k2pwn\") pod \"error-404-isvc-8eede-predictor-5d489c9599-98jkr\" (UID: \"73a12907-2862-4ed2-9a2b-ef500261948f\") " pod="kserve-ci-e2e-test/error-404-isvc-8eede-predictor-5d489c9599-98jkr"
Sep 17 18:02:36.625618 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:02:36.625512    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/73a12907-2862-4ed2-9a2b-ef500261948f-proxy-tls\") pod \"error-404-isvc-8eede-predictor-5d489c9599-98jkr\" (UID: \"73a12907-2862-4ed2-9a2b-ef500261948f\") " pod="kserve-ci-e2e-test/error-404-isvc-8eede-predictor-5d489c9599-98jkr"
Sep 17 18:02:36.626007 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:02:36.625620    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/73a12907-2862-4ed2-9a2b-ef500261948f-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-8eede-predictor-5d489c9599-98jkr\" (UID: \"73a12907-2862-4ed2-9a2b-ef500261948f\") " pod="kserve-ci-e2e-test/error-404-isvc-8eede-predictor-5d489c9599-98jkr"
Sep 17 18:02:36.626007 ip-10-0-139-237 kubenswrapper[2567]: E0917 18:02:36.625652    2567 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-8eede-predictor-serving-cert: secret "error-404-isvc-8eede-predictor-serving-cert" not found
Sep 17 18:02:36.626007 ip-10-0-139-237 kubenswrapper[2567]: E0917 18:02:36.625714    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/73a12907-2862-4ed2-9a2b-ef500261948f-proxy-tls podName:73a12907-2862-4ed2-9a2b-ef500261948f nodeName:}" failed. No retries permitted until 2026-09-17 18:02:37.125698678 +0000 UTC m=+1298.149303796 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/73a12907-2862-4ed2-9a2b-ef500261948f-proxy-tls") pod "error-404-isvc-8eede-predictor-5d489c9599-98jkr" (UID: "73a12907-2862-4ed2-9a2b-ef500261948f") : secret "error-404-isvc-8eede-predictor-serving-cert" not found
Sep 17 18:02:36.626007 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:02:36.625656    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2pwn\" (UniqueName: \"kubernetes.io/projected/73a12907-2862-4ed2-9a2b-ef500261948f-kube-api-access-k2pwn\") pod \"error-404-isvc-8eede-predictor-5d489c9599-98jkr\" (UID: \"73a12907-2862-4ed2-9a2b-ef500261948f\") " pod="kserve-ci-e2e-test/error-404-isvc-8eede-predictor-5d489c9599-98jkr"
Sep 17 18:02:36.626301 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:02:36.626284    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/73a12907-2862-4ed2-9a2b-ef500261948f-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-8eede-predictor-5d489c9599-98jkr\" (UID: \"73a12907-2862-4ed2-9a2b-ef500261948f\") " pod="kserve-ci-e2e-test/error-404-isvc-8eede-predictor-5d489c9599-98jkr"
Sep 17 18:02:36.636177 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:02:36.636154    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k2pwn\" (UniqueName: \"kubernetes.io/projected/73a12907-2862-4ed2-9a2b-ef500261948f-kube-api-access-k2pwn\") pod \"error-404-isvc-8eede-predictor-5d489c9599-98jkr\" (UID: \"73a12907-2862-4ed2-9a2b-ef500261948f\") " pod="kserve-ci-e2e-test/error-404-isvc-8eede-predictor-5d489c9599-98jkr"
Sep 17 18:02:36.767795 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:02:36.767763    2567 generic.go:356] "Generic (PLEG): container finished" podID="74c1ac49-8a02-45ec-b881-59bbb009d858" containerID="c6290751b1a7129c132628a82b06856ce94926f3149f6273c38b4d5cb8b039fa" exitCode=2
Sep 17 18:02:36.767961 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:02:36.767842    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-8649f-predictor-f69888d4c-dvxpd" event={"ID":"74c1ac49-8a02-45ec-b881-59bbb009d858","Type":"ContainerDied","Data":"c6290751b1a7129c132628a82b06856ce94926f3149f6273c38b4d5cb8b039fa"}
Sep 17 18:02:36.863826 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:02:36.863795    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 18:02:36.964557 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:02:36.964470    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 18:02:37.129329 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:02:37.129296    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/73a12907-2862-4ed2-9a2b-ef500261948f-proxy-tls\") pod \"error-404-isvc-8eede-predictor-5d489c9599-98jkr\" (UID: \"73a12907-2862-4ed2-9a2b-ef500261948f\") " pod="kserve-ci-e2e-test/error-404-isvc-8eede-predictor-5d489c9599-98jkr"
Sep 17 18:02:37.131675 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:02:37.131659    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/73a12907-2862-4ed2-9a2b-ef500261948f-proxy-tls\") pod \"error-404-isvc-8eede-predictor-5d489c9599-98jkr\" (UID: \"73a12907-2862-4ed2-9a2b-ef500261948f\") " pod="kserve-ci-e2e-test/error-404-isvc-8eede-predictor-5d489c9599-98jkr"
Sep 17 18:02:37.329495 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:02:37.329470    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-8eede-predictor-5d489c9599-98jkr"
Sep 17 18:02:37.619342 ip-10-0-139-237 kubenswrapper[2567]: W0917 18:02:37.619313    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73a12907_2862_4ed2_9a2b_ef500261948f.slice/crio-bcc404ba333179e2492066e101036f2e415346112ffd0eb167ef5e05002c11f7 WatchSource:0}: Error finding container bcc404ba333179e2492066e101036f2e415346112ffd0eb167ef5e05002c11f7: Status 404 returned error can't find the container with id bcc404ba333179e2492066e101036f2e415346112ffd0eb167ef5e05002c11f7
Sep 17 18:02:37.619479 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:02:37.619455    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-8eede-predictor-5d489c9599-98jkr"]
Sep 17 18:02:37.774301 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:02:37.774265    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-8eede-predictor-5d489c9599-98jkr" event={"ID":"73a12907-2862-4ed2-9a2b-ef500261948f","Type":"ContainerStarted","Data":"81498851ff8cb7eaafaf0a477592765ecf452c763ccd20ad51756226e3763480"}
Sep 17 18:02:37.774683 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:02:37.774336    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-8eede-predictor-5d489c9599-98jkr" event={"ID":"73a12907-2862-4ed2-9a2b-ef500261948f","Type":"ContainerStarted","Data":"1f2c131eb3a75fa2fb6cbd107a5fd523700ab864e0a84882d992d221302ab329"}
Sep 17 18:02:37.774683 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:02:37.774368    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-8eede-predictor-5d489c9599-98jkr"
Sep 17 18:02:37.774683 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:02:37.774399    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-8eede-predictor-5d489c9599-98jkr" event={"ID":"73a12907-2862-4ed2-9a2b-ef500261948f","Type":"ContainerStarted","Data":"bcc404ba333179e2492066e101036f2e415346112ffd0eb167ef5e05002c11f7"}
Sep 17 18:02:37.793801 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:02:37.793757    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-8eede-predictor-5d489c9599-98jkr" podStartSLOduration=1.793740454 podStartE2EDuration="1.793740454s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:02:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:02:37.792298207 +0000 UTC m=+1298.815903368" watchObservedRunningTime="2026-09-17 18:02:37.793740454 +0000 UTC m=+1298.817345651"
Sep 17 18:02:38.470998 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:02:38.470954    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-8649f-predictor-f69888d4c-dvxpd" podUID="74c1ac49-8a02-45ec-b881-59bbb009d858" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.33:8643/healthz\": dial tcp 10.134.0.33:8643: connect: connection refused"
Sep 17 18:02:38.739394 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:02:38.739309    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-8649f-fc945575-t7558" podUID="34575ee3-56a0-46aa-9055-7f1dbd70b55f" containerName="sequence-graph-8649f" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 18:02:38.778182 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:02:38.778150    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-8eede-predictor-5d489c9599-98jkr"
Sep 17 18:02:38.779338 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:02:38.779311    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-8eede-predictor-5d489c9599-98jkr" podUID="73a12907-2862-4ed2-9a2b-ef500261948f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Sep 17 18:02:39.362998 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:02:39.362972    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-8649f-predictor-f69888d4c-dvxpd"
Sep 17 18:02:39.448699 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:02:39.448631    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/74c1ac49-8a02-45ec-b881-59bbb009d858-proxy-tls\") pod \"74c1ac49-8a02-45ec-b881-59bbb009d858\" (UID: \"74c1ac49-8a02-45ec-b881-59bbb009d858\") "
Sep 17 18:02:39.448699 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:02:39.448687    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vkmlp\" (UniqueName: \"kubernetes.io/projected/74c1ac49-8a02-45ec-b881-59bbb009d858-kube-api-access-vkmlp\") pod \"74c1ac49-8a02-45ec-b881-59bbb009d858\" (UID: \"74c1ac49-8a02-45ec-b881-59bbb009d858\") "
Sep 17 18:02:39.448914 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:02:39.448709    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/74c1ac49-8a02-45ec-b881-59bbb009d858-kube-rbac-proxy-sar-config\") pod \"74c1ac49-8a02-45ec-b881-59bbb009d858\" (UID: \"74c1ac49-8a02-45ec-b881-59bbb009d858\") "
Sep 17 18:02:39.449096 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:02:39.449071    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74c1ac49-8a02-45ec-b881-59bbb009d858-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "74c1ac49-8a02-45ec-b881-59bbb009d858" (UID: "74c1ac49-8a02-45ec-b881-59bbb009d858"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:02:39.450817 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:02:39.450790    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/74c1ac49-8a02-45ec-b881-59bbb009d858-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "74c1ac49-8a02-45ec-b881-59bbb009d858" (UID: "74c1ac49-8a02-45ec-b881-59bbb009d858"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:02:39.450946 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:02:39.450822    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74c1ac49-8a02-45ec-b881-59bbb009d858-kube-api-access-vkmlp" (OuterVolumeSpecName: "kube-api-access-vkmlp") pod "74c1ac49-8a02-45ec-b881-59bbb009d858" (UID: "74c1ac49-8a02-45ec-b881-59bbb009d858"). InnerVolumeSpecName "kube-api-access-vkmlp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:02:39.550054 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:02:39.550013    2567 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/74c1ac49-8a02-45ec-b881-59bbb009d858-proxy-tls\") on node \"ip-10-0-139-237.ec2.internal\" DevicePath \"\""
Sep 17 18:02:39.550054 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:02:39.550049    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vkmlp\" (UniqueName: \"kubernetes.io/projected/74c1ac49-8a02-45ec-b881-59bbb009d858-kube-api-access-vkmlp\") on node \"ip-10-0-139-237.ec2.internal\" DevicePath \"\""
Sep 17 18:02:39.550054 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:02:39.550058    2567 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/74c1ac49-8a02-45ec-b881-59bbb009d858-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-237.ec2.internal\" DevicePath \"\""
Sep 17 18:02:39.783838 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:02:39.783756    2567 generic.go:356] "Generic (PLEG): container finished" podID="74c1ac49-8a02-45ec-b881-59bbb009d858" containerID="848fd758bfc502873a4860495a96c1f00150c691a8c6e9d8da7d66d0a0093f16" exitCode=0
Sep 17 18:02:39.783838 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:02:39.783810    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-8649f-predictor-f69888d4c-dvxpd"
Sep 17 18:02:39.784287 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:02:39.783839    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-8649f-predictor-f69888d4c-dvxpd" event={"ID":"74c1ac49-8a02-45ec-b881-59bbb009d858","Type":"ContainerDied","Data":"848fd758bfc502873a4860495a96c1f00150c691a8c6e9d8da7d66d0a0093f16"}
Sep 17 18:02:39.784287 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:02:39.783906    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-8649f-predictor-f69888d4c-dvxpd" event={"ID":"74c1ac49-8a02-45ec-b881-59bbb009d858","Type":"ContainerDied","Data":"2c3503bba228a1280d6c89cab8342efc7fb748c63f722f0bf13b1f3713ee74b9"}
Sep 17 18:02:39.784287 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:02:39.783945    2567 scope.go:117] "RemoveContainer" containerID="c6290751b1a7129c132628a82b06856ce94926f3149f6273c38b4d5cb8b039fa"
Sep 17 18:02:39.784452 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:02:39.784413    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-8eede-predictor-5d489c9599-98jkr" podUID="73a12907-2862-4ed2-9a2b-ef500261948f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Sep 17 18:02:39.792660 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:02:39.792643    2567 scope.go:117] "RemoveContainer" containerID="848fd758bfc502873a4860495a96c1f00150c691a8c6e9d8da7d66d0a0093f16"
Sep 17 18:02:39.799809 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:02:39.799788    2567 scope.go:117] "RemoveContainer" containerID="c6290751b1a7129c132628a82b06856ce94926f3149f6273c38b4d5cb8b039fa"
Sep 17 18:02:39.800062 ip-10-0-139-237 kubenswrapper[2567]: E0917 18:02:39.800045    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c6290751b1a7129c132628a82b06856ce94926f3149f6273c38b4d5cb8b039fa\": container with ID starting with c6290751b1a7129c132628a82b06856ce94926f3149f6273c38b4d5cb8b039fa not found: ID does not exist" containerID="c6290751b1a7129c132628a82b06856ce94926f3149f6273c38b4d5cb8b039fa"
Sep 17 18:02:39.800113 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:02:39.800070    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c6290751b1a7129c132628a82b06856ce94926f3149f6273c38b4d5cb8b039fa"} err="failed to get container status \"c6290751b1a7129c132628a82b06856ce94926f3149f6273c38b4d5cb8b039fa\": rpc error: code = NotFound desc = could not find container \"c6290751b1a7129c132628a82b06856ce94926f3149f6273c38b4d5cb8b039fa\": container with ID starting with c6290751b1a7129c132628a82b06856ce94926f3149f6273c38b4d5cb8b039fa not found: ID does not exist"
Sep 17 18:02:39.800113 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:02:39.800086    2567 scope.go:117] "RemoveContainer" containerID="848fd758bfc502873a4860495a96c1f00150c691a8c6e9d8da7d66d0a0093f16"
Sep 17 18:02:39.800343 ip-10-0-139-237 kubenswrapper[2567]: E0917 18:02:39.800325    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"848fd758bfc502873a4860495a96c1f00150c691a8c6e9d8da7d66d0a0093f16\": container with ID starting with 848fd758bfc502873a4860495a96c1f00150c691a8c6e9d8da7d66d0a0093f16 not found: ID does not exist" containerID="848fd758bfc502873a4860495a96c1f00150c691a8c6e9d8da7d66d0a0093f16"
Sep 17 18:02:39.800401 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:02:39.800348    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"848fd758bfc502873a4860495a96c1f00150c691a8c6e9d8da7d66d0a0093f16"} err="failed to get container status \"848fd758bfc502873a4860495a96c1f00150c691a8c6e9d8da7d66d0a0093f16\": rpc error: code = NotFound desc = could not find container \"848fd758bfc502873a4860495a96c1f00150c691a8c6e9d8da7d66d0a0093f16\": container with ID starting with 848fd758bfc502873a4860495a96c1f00150c691a8c6e9d8da7d66d0a0093f16 not found: ID does not exist"
Sep 17 18:02:39.808115 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:02:39.808095    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-8649f-predictor-f69888d4c-dvxpd"]
Sep 17 18:02:39.815952 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:02:39.815933    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-8649f-predictor-f69888d4c-dvxpd"]
Sep 17 18:02:41.286426 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:02:41.286394    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="74c1ac49-8a02-45ec-b881-59bbb009d858" path="/var/lib/kubelet/pods/74c1ac49-8a02-45ec-b881-59bbb009d858/volumes"
Sep 17 18:02:43.741366 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:02:43.741330    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-8649f-fc945575-t7558" podUID="34575ee3-56a0-46aa-9055-7f1dbd70b55f" containerName="sequence-graph-8649f" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 18:02:44.789099 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:02:44.789068    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-8eede-predictor-5d489c9599-98jkr"
Sep 17 18:02:44.789558 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:02:44.789510    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-8eede-predictor-5d489c9599-98jkr" podUID="73a12907-2862-4ed2-9a2b-ef500261948f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Sep 17 18:02:46.171888 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:02:46.171857    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-9d702-6849cf5859-m6bk5"]
Sep 17 18:02:46.172254 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:02:46.172189    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="74c1ac49-8a02-45ec-b881-59bbb009d858" containerName="kube-rbac-proxy"
Sep 17 18:02:46.172254 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:02:46.172199    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="74c1ac49-8a02-45ec-b881-59bbb009d858" containerName="kube-rbac-proxy"
Sep 17 18:02:46.172254 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:02:46.172218    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="74c1ac49-8a02-45ec-b881-59bbb009d858" containerName="kserve-container"
Sep 17 18:02:46.172254 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:02:46.172223    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="74c1ac49-8a02-45ec-b881-59bbb009d858" containerName="kserve-container"
Sep 17 18:02:46.172378 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:02:46.172268    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="74c1ac49-8a02-45ec-b881-59bbb009d858" containerName="kserve-container"
Sep 17 18:02:46.172378 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:02:46.172275    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="74c1ac49-8a02-45ec-b881-59bbb009d858" containerName="kube-rbac-proxy"
Sep 17 18:02:46.175210 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:02:46.175190    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-9d702-6849cf5859-m6bk5"
Sep 17 18:02:46.177650 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:02:46.177627    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"ensemble-graph-9d702-serving-cert\""
Sep 17 18:02:46.177729 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:02:46.177655    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"ensemble-graph-9d702-kube-rbac-proxy-sar-config\""
Sep 17 18:02:46.180434 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:02:46.180408    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 18:02:46.185151 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:02:46.185128    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-9d702-6849cf5859-m6bk5"]
Sep 17 18:02:46.306954 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:02:46.306923    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e5beaf37-1f94-4f07-8e24-2c92390ec75c-proxy-tls\") pod \"ensemble-graph-9d702-6849cf5859-m6bk5\" (UID: \"e5beaf37-1f94-4f07-8e24-2c92390ec75c\") " pod="kserve-ci-e2e-test/ensemble-graph-9d702-6849cf5859-m6bk5"
Sep 17 18:02:46.307113 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:02:46.306986    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e5beaf37-1f94-4f07-8e24-2c92390ec75c-openshift-service-ca-bundle\") pod \"ensemble-graph-9d702-6849cf5859-m6bk5\" (UID: \"e5beaf37-1f94-4f07-8e24-2c92390ec75c\") " pod="kserve-ci-e2e-test/ensemble-graph-9d702-6849cf5859-m6bk5"
Sep 17 18:02:46.407626 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:02:46.407586    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e5beaf37-1f94-4f07-8e24-2c92390ec75c-proxy-tls\") pod \"ensemble-graph-9d702-6849cf5859-m6bk5\" (UID: \"e5beaf37-1f94-4f07-8e24-2c92390ec75c\") " pod="kserve-ci-e2e-test/ensemble-graph-9d702-6849cf5859-m6bk5"
Sep 17 18:02:46.407810 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:02:46.407677    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e5beaf37-1f94-4f07-8e24-2c92390ec75c-openshift-service-ca-bundle\") pod \"ensemble-graph-9d702-6849cf5859-m6bk5\" (UID: \"e5beaf37-1f94-4f07-8e24-2c92390ec75c\") " pod="kserve-ci-e2e-test/ensemble-graph-9d702-6849cf5859-m6bk5"
Sep 17 18:02:46.407810 ip-10-0-139-237 kubenswrapper[2567]: E0917 18:02:46.407737    2567 secret.go:189] Couldn't get secret kserve-ci-e2e-test/ensemble-graph-9d702-serving-cert: secret "ensemble-graph-9d702-serving-cert" not found
Sep 17 18:02:46.407911 ip-10-0-139-237 kubenswrapper[2567]: E0917 18:02:46.407816    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e5beaf37-1f94-4f07-8e24-2c92390ec75c-proxy-tls podName:e5beaf37-1f94-4f07-8e24-2c92390ec75c nodeName:}" failed. No retries permitted until 2026-09-17 18:02:46.907798665 +0000 UTC m=+1307.931403779 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/e5beaf37-1f94-4f07-8e24-2c92390ec75c-proxy-tls") pod "ensemble-graph-9d702-6849cf5859-m6bk5" (UID: "e5beaf37-1f94-4f07-8e24-2c92390ec75c") : secret "ensemble-graph-9d702-serving-cert" not found
Sep 17 18:02:46.408251 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:02:46.408232    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e5beaf37-1f94-4f07-8e24-2c92390ec75c-openshift-service-ca-bundle\") pod \"ensemble-graph-9d702-6849cf5859-m6bk5\" (UID: \"e5beaf37-1f94-4f07-8e24-2c92390ec75c\") " pod="kserve-ci-e2e-test/ensemble-graph-9d702-6849cf5859-m6bk5"
Sep 17 18:02:46.805077 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:02:46.805049    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 18:02:46.912141 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:02:46.912100    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e5beaf37-1f94-4f07-8e24-2c92390ec75c-proxy-tls\") pod \"ensemble-graph-9d702-6849cf5859-m6bk5\" (UID: \"e5beaf37-1f94-4f07-8e24-2c92390ec75c\") " pod="kserve-ci-e2e-test/ensemble-graph-9d702-6849cf5859-m6bk5"
Sep 17 18:02:46.914499 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:02:46.914468    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e5beaf37-1f94-4f07-8e24-2c92390ec75c-proxy-tls\") pod \"ensemble-graph-9d702-6849cf5859-m6bk5\" (UID: \"e5beaf37-1f94-4f07-8e24-2c92390ec75c\") " pod="kserve-ci-e2e-test/ensemble-graph-9d702-6849cf5859-m6bk5"
Sep 17 18:02:47.086101 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:02:47.086018    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-9d702-6849cf5859-m6bk5"
Sep 17 18:02:47.358072 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:02:47.358038    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-9d702-6849cf5859-m6bk5"]
Sep 17 18:02:47.360934 ip-10-0-139-237 kubenswrapper[2567]: W0917 18:02:47.360909    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5beaf37_1f94_4f07_8e24_2c92390ec75c.slice/crio-fed0ba5b0f8ce06d8cf91ac7507a60c0a54643247cbb57dca8b85660c7eb6e80 WatchSource:0}: Error finding container fed0ba5b0f8ce06d8cf91ac7507a60c0a54643247cbb57dca8b85660c7eb6e80: Status 404 returned error can't find the container with id fed0ba5b0f8ce06d8cf91ac7507a60c0a54643247cbb57dca8b85660c7eb6e80
Sep 17 18:02:47.818169 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:02:47.818138    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-9d702-6849cf5859-m6bk5" event={"ID":"e5beaf37-1f94-4f07-8e24-2c92390ec75c","Type":"ContainerStarted","Data":"371541d0a024a2041ba16c804dac22058f22c660e795ec8b106d6e97baf8bf4d"}
Sep 17 18:02:47.818349 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:02:47.818190    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-9d702-6849cf5859-m6bk5" event={"ID":"e5beaf37-1f94-4f07-8e24-2c92390ec75c","Type":"ContainerStarted","Data":"fed0ba5b0f8ce06d8cf91ac7507a60c0a54643247cbb57dca8b85660c7eb6e80"}
Sep 17 18:02:47.818349 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:02:47.818218    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/ensemble-graph-9d702-6849cf5859-m6bk5"
Sep 17 18:02:47.835817 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:02:47.835769    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/ensemble-graph-9d702-6849cf5859-m6bk5" podStartSLOduration=1.835753044 podStartE2EDuration="1.835753044s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:02:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:02:47.834415249 +0000 UTC m=+1308.858020387" watchObservedRunningTime="2026-09-17 18:02:47.835753044 +0000 UTC m=+1308.859358177"
Sep 17 18:02:48.740210 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:02:48.740174    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-8649f-fc945575-t7558" podUID="34575ee3-56a0-46aa-9055-7f1dbd70b55f" containerName="sequence-graph-8649f" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 18:02:48.740616 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:02:48.740273    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-8649f-fc945575-t7558"
Sep 17 18:02:53.739434 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:02:53.739391    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-8649f-fc945575-t7558" podUID="34575ee3-56a0-46aa-9055-7f1dbd70b55f" containerName="sequence-graph-8649f" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 18:02:53.827646 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:02:53.827613    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/ensemble-graph-9d702-6849cf5859-m6bk5"
Sep 17 18:02:54.790215 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:02:54.790172    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-8eede-predictor-5d489c9599-98jkr" podUID="73a12907-2862-4ed2-9a2b-ef500261948f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Sep 17 18:02:58.739449 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:02:58.739388    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-8649f-fc945575-t7558" podUID="34575ee3-56a0-46aa-9055-7f1dbd70b55f" containerName="sequence-graph-8649f" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 18:03:03.739450 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:03:03.739413    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-8649f-fc945575-t7558" podUID="34575ee3-56a0-46aa-9055-7f1dbd70b55f" containerName="sequence-graph-8649f" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 18:03:04.789462 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:03:04.789423    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-8eede-predictor-5d489c9599-98jkr" podUID="73a12907-2862-4ed2-9a2b-ef500261948f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Sep 17 18:03:06.226567 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:03:06.226518    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-8649f-fc945575-t7558"
Sep 17 18:03:06.383881 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:03:06.383802    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/34575ee3-56a0-46aa-9055-7f1dbd70b55f-proxy-tls\") pod \"34575ee3-56a0-46aa-9055-7f1dbd70b55f\" (UID: \"34575ee3-56a0-46aa-9055-7f1dbd70b55f\") "
Sep 17 18:03:06.383881 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:03:06.383855    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/34575ee3-56a0-46aa-9055-7f1dbd70b55f-openshift-service-ca-bundle\") pod \"34575ee3-56a0-46aa-9055-7f1dbd70b55f\" (UID: \"34575ee3-56a0-46aa-9055-7f1dbd70b55f\") "
Sep 17 18:03:06.384260 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:03:06.384235    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34575ee3-56a0-46aa-9055-7f1dbd70b55f-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "34575ee3-56a0-46aa-9055-7f1dbd70b55f" (UID: "34575ee3-56a0-46aa-9055-7f1dbd70b55f"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:03:06.385857 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:03:06.385827    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/34575ee3-56a0-46aa-9055-7f1dbd70b55f-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "34575ee3-56a0-46aa-9055-7f1dbd70b55f" (UID: "34575ee3-56a0-46aa-9055-7f1dbd70b55f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:03:06.485158 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:03:06.485125    2567 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/34575ee3-56a0-46aa-9055-7f1dbd70b55f-proxy-tls\") on node \"ip-10-0-139-237.ec2.internal\" DevicePath \"\""
Sep 17 18:03:06.485158 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:03:06.485152    2567 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/34575ee3-56a0-46aa-9055-7f1dbd70b55f-openshift-service-ca-bundle\") on node \"ip-10-0-139-237.ec2.internal\" DevicePath \"\""
Sep 17 18:03:06.898345 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:03:06.898312    2567 generic.go:356] "Generic (PLEG): container finished" podID="34575ee3-56a0-46aa-9055-7f1dbd70b55f" containerID="e9a18029f62ad844a5df64aa95d9abee3e9dae3c8f7ab9299ba7f38fe1e4fb00" exitCode=0
Sep 17 18:03:06.898511 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:03:06.898356    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-8649f-fc945575-t7558"
Sep 17 18:03:06.898511 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:03:06.898382    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-8649f-fc945575-t7558" event={"ID":"34575ee3-56a0-46aa-9055-7f1dbd70b55f","Type":"ContainerDied","Data":"e9a18029f62ad844a5df64aa95d9abee3e9dae3c8f7ab9299ba7f38fe1e4fb00"}
Sep 17 18:03:06.898511 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:03:06.898423    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-8649f-fc945575-t7558" event={"ID":"34575ee3-56a0-46aa-9055-7f1dbd70b55f","Type":"ContainerDied","Data":"b0aa9014ae3de90d59fe665df15fc4fcde578f46839af3262c31f98988fc6dc7"}
Sep 17 18:03:06.898511 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:03:06.898447    2567 scope.go:117] "RemoveContainer" containerID="e9a18029f62ad844a5df64aa95d9abee3e9dae3c8f7ab9299ba7f38fe1e4fb00"
Sep 17 18:03:06.907281 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:03:06.907262    2567 scope.go:117] "RemoveContainer" containerID="e9a18029f62ad844a5df64aa95d9abee3e9dae3c8f7ab9299ba7f38fe1e4fb00"
Sep 17 18:03:06.907564 ip-10-0-139-237 kubenswrapper[2567]: E0917 18:03:06.907523    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e9a18029f62ad844a5df64aa95d9abee3e9dae3c8f7ab9299ba7f38fe1e4fb00\": container with ID starting with e9a18029f62ad844a5df64aa95d9abee3e9dae3c8f7ab9299ba7f38fe1e4fb00 not found: ID does not exist" containerID="e9a18029f62ad844a5df64aa95d9abee3e9dae3c8f7ab9299ba7f38fe1e4fb00"
Sep 17 18:03:06.907631 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:03:06.907579    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9a18029f62ad844a5df64aa95d9abee3e9dae3c8f7ab9299ba7f38fe1e4fb00"} err="failed to get container status \"e9a18029f62ad844a5df64aa95d9abee3e9dae3c8f7ab9299ba7f38fe1e4fb00\": rpc error: code = NotFound desc = could not find container \"e9a18029f62ad844a5df64aa95d9abee3e9dae3c8f7ab9299ba7f38fe1e4fb00\": container with ID starting with e9a18029f62ad844a5df64aa95d9abee3e9dae3c8f7ab9299ba7f38fe1e4fb00 not found: ID does not exist"
Sep 17 18:03:06.921248 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:03:06.921217    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-8649f-fc945575-t7558"]
Sep 17 18:03:06.927561 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:03:06.927523    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-8649f-fc945575-t7558"]
Sep 17 18:03:07.285946 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:03:07.285872    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="34575ee3-56a0-46aa-9055-7f1dbd70b55f" path="/var/lib/kubelet/pods/34575ee3-56a0-46aa-9055-7f1dbd70b55f/volumes"
Sep 17 18:03:14.789994 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:03:14.789915    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-8eede-predictor-5d489c9599-98jkr" podUID="73a12907-2862-4ed2-9a2b-ef500261948f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Sep 17 18:03:24.790327 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:03:24.790297    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-8eede-predictor-5d489c9599-98jkr"
Sep 17 18:03:36.356391 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:03:36.356358    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/sequence-graph-8eede-bdd8448d-m85hx"]
Sep 17 18:03:36.356836 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:03:36.356706    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="34575ee3-56a0-46aa-9055-7f1dbd70b55f" containerName="sequence-graph-8649f"
Sep 17 18:03:36.356836 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:03:36.356718    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="34575ee3-56a0-46aa-9055-7f1dbd70b55f" containerName="sequence-graph-8649f"
Sep 17 18:03:36.356836 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:03:36.356803    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="34575ee3-56a0-46aa-9055-7f1dbd70b55f" containerName="sequence-graph-8649f"
Sep 17 18:03:36.359356 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:03:36.359339    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-8eede-bdd8448d-m85hx"
Sep 17 18:03:36.363030 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:03:36.363003    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-8eede-kube-rbac-proxy-sar-config\""
Sep 17 18:03:36.363159 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:03:36.363003    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-8eede-serving-cert\""
Sep 17 18:03:36.367278 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:03:36.367253    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 18:03:36.370631 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:03:36.370598    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-8eede-bdd8448d-m85hx"]
Sep 17 18:03:36.432008 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:03:36.431974    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a4197f5d-92b4-44b7-a820-e3d30f8882a4-openshift-service-ca-bundle\") pod \"sequence-graph-8eede-bdd8448d-m85hx\" (UID: \"a4197f5d-92b4-44b7-a820-e3d30f8882a4\") " pod="kserve-ci-e2e-test/sequence-graph-8eede-bdd8448d-m85hx"
Sep 17 18:03:36.432146 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:03:36.432058    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a4197f5d-92b4-44b7-a820-e3d30f8882a4-proxy-tls\") pod \"sequence-graph-8eede-bdd8448d-m85hx\" (UID: \"a4197f5d-92b4-44b7-a820-e3d30f8882a4\") " pod="kserve-ci-e2e-test/sequence-graph-8eede-bdd8448d-m85hx"
Sep 17 18:03:36.533370 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:03:36.533337    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a4197f5d-92b4-44b7-a820-e3d30f8882a4-openshift-service-ca-bundle\") pod \"sequence-graph-8eede-bdd8448d-m85hx\" (UID: \"a4197f5d-92b4-44b7-a820-e3d30f8882a4\") " pod="kserve-ci-e2e-test/sequence-graph-8eede-bdd8448d-m85hx"
Sep 17 18:03:36.533556 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:03:36.533397    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a4197f5d-92b4-44b7-a820-e3d30f8882a4-proxy-tls\") pod \"sequence-graph-8eede-bdd8448d-m85hx\" (UID: \"a4197f5d-92b4-44b7-a820-e3d30f8882a4\") " pod="kserve-ci-e2e-test/sequence-graph-8eede-bdd8448d-m85hx"
Sep 17 18:03:36.534036 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:03:36.534014    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a4197f5d-92b4-44b7-a820-e3d30f8882a4-openshift-service-ca-bundle\") pod \"sequence-graph-8eede-bdd8448d-m85hx\" (UID: \"a4197f5d-92b4-44b7-a820-e3d30f8882a4\") " pod="kserve-ci-e2e-test/sequence-graph-8eede-bdd8448d-m85hx"
Sep 17 18:03:36.535724 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:03:36.535706    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a4197f5d-92b4-44b7-a820-e3d30f8882a4-proxy-tls\") pod \"sequence-graph-8eede-bdd8448d-m85hx\" (UID: \"a4197f5d-92b4-44b7-a820-e3d30f8882a4\") " pod="kserve-ci-e2e-test/sequence-graph-8eede-bdd8448d-m85hx"
Sep 17 18:03:36.670603 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:03:36.670494    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-8eede-bdd8448d-m85hx"
Sep 17 18:03:36.941607 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:03:36.941584    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-8eede-bdd8448d-m85hx"]
Sep 17 18:03:36.943812 ip-10-0-139-237 kubenswrapper[2567]: W0917 18:03:36.943782    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4197f5d_92b4_44b7_a820_e3d30f8882a4.slice/crio-152d090559b46a13c0a988e96ae31fa25fb6cbca23582dad7b717726885fe589 WatchSource:0}: Error finding container 152d090559b46a13c0a988e96ae31fa25fb6cbca23582dad7b717726885fe589: Status 404 returned error can't find the container with id 152d090559b46a13c0a988e96ae31fa25fb6cbca23582dad7b717726885fe589
Sep 17 18:03:37.025545 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:03:37.025501    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-8eede-bdd8448d-m85hx" event={"ID":"a4197f5d-92b4-44b7-a820-e3d30f8882a4","Type":"ContainerStarted","Data":"6459bdc4048b6f2a7445938daaa85d262475dfb1e48687d9e827068d93ae4d0d"}
Sep 17 18:03:37.025692 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:03:37.025583    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-8eede-bdd8448d-m85hx"
Sep 17 18:03:37.025692 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:03:37.025611    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-8eede-bdd8448d-m85hx" event={"ID":"a4197f5d-92b4-44b7-a820-e3d30f8882a4","Type":"ContainerStarted","Data":"152d090559b46a13c0a988e96ae31fa25fb6cbca23582dad7b717726885fe589"}
Sep 17 18:03:37.046285 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:03:37.046236    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/sequence-graph-8eede-bdd8448d-m85hx" podStartSLOduration=1.046217266 podStartE2EDuration="1.046217266s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:03:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:03:37.045857782 +0000 UTC m=+1358.069462914" watchObservedRunningTime="2026-09-17 18:03:37.046217266 +0000 UTC m=+1358.069822403"
Sep 17 18:03:37.735852 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:03:37.735817    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 18:03:43.034480 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:03:43.034450    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/sequence-graph-8eede-bdd8448d-m85hx"
Sep 17 18:07:04.915326 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:07:04.915279    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 18:11:00.786286 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:11:00.786252    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-9d702-6849cf5859-m6bk5"]
Sep 17 18:11:00.786844 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:11:00.786504    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/ensemble-graph-9d702-6849cf5859-m6bk5" podUID="e5beaf37-1f94-4f07-8e24-2c92390ec75c" containerName="ensemble-graph-9d702" containerID="cri-o://371541d0a024a2041ba16c804dac22058f22c660e795ec8b106d6e97baf8bf4d" gracePeriod=30
Sep 17 18:11:00.847447 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:11:00.847412    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 18:11:00.972086 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:11:00.972057    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-9d702-predictor-5967c4d597-nkqp8"]
Sep 17 18:11:00.972318 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:11:00.972286    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-9d702-predictor-5967c4d597-nkqp8" podUID="f161c155-a751-4207-a686-6154fb8949a9" containerName="kserve-container" containerID="cri-o://e63913250ccc577d9238c3f4d7ad26b296c597759a6968a164fec61a9f3bb50e" gracePeriod=30
Sep 17 18:11:00.972476 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:11:00.972328    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-9d702-predictor-5967c4d597-nkqp8" podUID="f161c155-a751-4207-a686-6154fb8949a9" containerName="kube-rbac-proxy" containerID="cri-o://e91634bff0c24472ab73401dd9d8909baf410768c552e1665de1269316314a01" gracePeriod=30
Sep 17 18:11:01.109653 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:11:01.109620    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-de19c-predictor-d5c8fcc89-47k8v"]
Sep 17 18:11:01.112881 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:11:01.112864    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-de19c-predictor-d5c8fcc89-47k8v"
Sep 17 18:11:01.115871 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:11:01.115850    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-de19c-predictor-serving-cert\""
Sep 17 18:11:01.115968 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:11:01.115853    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-de19c-kube-rbac-proxy-sar-config\""
Sep 17 18:11:01.124378 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:11:01.124355    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-de19c-predictor-d5c8fcc89-47k8v"]
Sep 17 18:11:01.220070 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:11:01.220038    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ce283b34-ac81-4d48-a2c1-a75f93b3958d-proxy-tls\") pod \"error-404-isvc-de19c-predictor-d5c8fcc89-47k8v\" (UID: \"ce283b34-ac81-4d48-a2c1-a75f93b3958d\") " pod="kserve-ci-e2e-test/error-404-isvc-de19c-predictor-d5c8fcc89-47k8v"
Sep 17 18:11:01.220229 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:11:01.220088    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ce283b34-ac81-4d48-a2c1-a75f93b3958d-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-de19c-predictor-d5c8fcc89-47k8v\" (UID: \"ce283b34-ac81-4d48-a2c1-a75f93b3958d\") " pod="kserve-ci-e2e-test/error-404-isvc-de19c-predictor-d5c8fcc89-47k8v"
Sep 17 18:11:01.220229 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:11:01.220136    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jsxfj\" (UniqueName: \"kubernetes.io/projected/ce283b34-ac81-4d48-a2c1-a75f93b3958d-kube-api-access-jsxfj\") pod \"error-404-isvc-de19c-predictor-d5c8fcc89-47k8v\" (UID: \"ce283b34-ac81-4d48-a2c1-a75f93b3958d\") " pod="kserve-ci-e2e-test/error-404-isvc-de19c-predictor-d5c8fcc89-47k8v"
Sep 17 18:11:01.321299 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:11:01.321266    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ce283b34-ac81-4d48-a2c1-a75f93b3958d-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-de19c-predictor-d5c8fcc89-47k8v\" (UID: \"ce283b34-ac81-4d48-a2c1-a75f93b3958d\") " pod="kserve-ci-e2e-test/error-404-isvc-de19c-predictor-d5c8fcc89-47k8v"
Sep 17 18:11:01.321450 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:11:01.321312    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jsxfj\" (UniqueName: \"kubernetes.io/projected/ce283b34-ac81-4d48-a2c1-a75f93b3958d-kube-api-access-jsxfj\") pod \"error-404-isvc-de19c-predictor-d5c8fcc89-47k8v\" (UID: \"ce283b34-ac81-4d48-a2c1-a75f93b3958d\") " pod="kserve-ci-e2e-test/error-404-isvc-de19c-predictor-d5c8fcc89-47k8v"
Sep 17 18:11:01.321450 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:11:01.321359    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ce283b34-ac81-4d48-a2c1-a75f93b3958d-proxy-tls\") pod \"error-404-isvc-de19c-predictor-d5c8fcc89-47k8v\" (UID: \"ce283b34-ac81-4d48-a2c1-a75f93b3958d\") " pod="kserve-ci-e2e-test/error-404-isvc-de19c-predictor-d5c8fcc89-47k8v"
Sep 17 18:11:01.326642 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:11:01.326618    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-de19c-kube-rbac-proxy-sar-config\""
Sep 17 18:11:01.326784 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:11:01.326671    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-de19c-predictor-serving-cert\""
Sep 17 18:11:01.331958 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:11:01.331934    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ce283b34-ac81-4d48-a2c1-a75f93b3958d-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-de19c-predictor-d5c8fcc89-47k8v\" (UID: \"ce283b34-ac81-4d48-a2c1-a75f93b3958d\") " pod="kserve-ci-e2e-test/error-404-isvc-de19c-predictor-d5c8fcc89-47k8v"
Sep 17 18:11:01.333664 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:11:01.333646    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ce283b34-ac81-4d48-a2c1-a75f93b3958d-proxy-tls\") pod \"error-404-isvc-de19c-predictor-d5c8fcc89-47k8v\" (UID: \"ce283b34-ac81-4d48-a2c1-a75f93b3958d\") " pod="kserve-ci-e2e-test/error-404-isvc-de19c-predictor-d5c8fcc89-47k8v"
Sep 17 18:11:01.342893 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:11:01.342832    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jsxfj\" (UniqueName: \"kubernetes.io/projected/ce283b34-ac81-4d48-a2c1-a75f93b3958d-kube-api-access-jsxfj\") pod \"error-404-isvc-de19c-predictor-d5c8fcc89-47k8v\" (UID: \"ce283b34-ac81-4d48-a2c1-a75f93b3958d\") " pod="kserve-ci-e2e-test/error-404-isvc-de19c-predictor-d5c8fcc89-47k8v"
Sep 17 18:11:01.423697 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:11:01.423667    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-de19c-predictor-d5c8fcc89-47k8v"
Sep 17 18:11:01.697518 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:11:01.697494    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-de19c-predictor-d5c8fcc89-47k8v"]
Sep 17 18:11:01.699851 ip-10-0-139-237 kubenswrapper[2567]: W0917 18:11:01.699821    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce283b34_ac81_4d48_a2c1_a75f93b3958d.slice/crio-c47cfeeea27101d294b438ecc70a34c080b3a209bd5dc0fef588ea26ccc3b862 WatchSource:0}: Error finding container c47cfeeea27101d294b438ecc70a34c080b3a209bd5dc0fef588ea26ccc3b862: Status 404 returned error can't find the container with id c47cfeeea27101d294b438ecc70a34c080b3a209bd5dc0fef588ea26ccc3b862
Sep 17 18:11:01.827292 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:11:01.827262    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-de19c-predictor-d5c8fcc89-47k8v" event={"ID":"ce283b34-ac81-4d48-a2c1-a75f93b3958d","Type":"ContainerStarted","Data":"99c80879680f9790fc8f2aba67a2ccbdcc331f218f4961fe9396cf57ac612323"}
Sep 17 18:11:01.827642 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:11:01.827318    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-de19c-predictor-d5c8fcc89-47k8v" event={"ID":"ce283b34-ac81-4d48-a2c1-a75f93b3958d","Type":"ContainerStarted","Data":"e005040450edc6bf68e69c712b5fc5d0277b17855d03da448bf73f90afe75a94"}
Sep 17 18:11:01.827642 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:11:01.827344    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-de19c-predictor-d5c8fcc89-47k8v"
Sep 17 18:11:01.827642 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:11:01.827369    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-de19c-predictor-d5c8fcc89-47k8v" event={"ID":"ce283b34-ac81-4d48-a2c1-a75f93b3958d","Type":"ContainerStarted","Data":"c47cfeeea27101d294b438ecc70a34c080b3a209bd5dc0fef588ea26ccc3b862"}
Sep 17 18:11:01.829002 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:11:01.828980    2567 generic.go:356] "Generic (PLEG): container finished" podID="f161c155-a751-4207-a686-6154fb8949a9" containerID="e91634bff0c24472ab73401dd9d8909baf410768c552e1665de1269316314a01" exitCode=2
Sep 17 18:11:01.829089 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:11:01.829046    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-9d702-predictor-5967c4d597-nkqp8" event={"ID":"f161c155-a751-4207-a686-6154fb8949a9","Type":"ContainerDied","Data":"e91634bff0c24472ab73401dd9d8909baf410768c552e1665de1269316314a01"}
Sep 17 18:11:01.849070 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:11:01.849018    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-de19c-predictor-d5c8fcc89-47k8v" podStartSLOduration=0.849003066 podStartE2EDuration="849.003066ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:11:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:11:01.846551862 +0000 UTC m=+1802.870157022" watchObservedRunningTime="2026-09-17 18:11:01.849003066 +0000 UTC m=+1802.872608227"
Sep 17 18:11:02.153176 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:11:02.153139    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 18:11:02.833324 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:11:02.833291    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-de19c-predictor-d5c8fcc89-47k8v"
Sep 17 18:11:02.834759 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:11:02.834736    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-de19c-predictor-d5c8fcc89-47k8v" podUID="ce283b34-ac81-4d48-a2c1-a75f93b3958d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Sep 17 18:11:03.826503 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:11:03.826462    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-9d702-6849cf5859-m6bk5" podUID="e5beaf37-1f94-4f07-8e24-2c92390ec75c" containerName="ensemble-graph-9d702" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 18:11:03.837380 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:11:03.837348    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-de19c-predictor-d5c8fcc89-47k8v" podUID="ce283b34-ac81-4d48-a2c1-a75f93b3958d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Sep 17 18:11:04.314788 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:11:04.314765    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-9d702-predictor-5967c4d597-nkqp8"
Sep 17 18:11:04.448101 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:11:04.448012    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f161c155-a751-4207-a686-6154fb8949a9-proxy-tls\") pod \"f161c155-a751-4207-a686-6154fb8949a9\" (UID: \"f161c155-a751-4207-a686-6154fb8949a9\") "
Sep 17 18:11:04.448282 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:11:04.448114    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x6fv8\" (UniqueName: \"kubernetes.io/projected/f161c155-a751-4207-a686-6154fb8949a9-kube-api-access-x6fv8\") pod \"f161c155-a751-4207-a686-6154fb8949a9\" (UID: \"f161c155-a751-4207-a686-6154fb8949a9\") "
Sep 17 18:11:04.448282 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:11:04.448133    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f161c155-a751-4207-a686-6154fb8949a9-kube-rbac-proxy-sar-config\") pod \"f161c155-a751-4207-a686-6154fb8949a9\" (UID: \"f161c155-a751-4207-a686-6154fb8949a9\") "
Sep 17 18:11:04.448566 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:11:04.448512    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f161c155-a751-4207-a686-6154fb8949a9-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "f161c155-a751-4207-a686-6154fb8949a9" (UID: "f161c155-a751-4207-a686-6154fb8949a9"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:11:04.450120 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:11:04.450090    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f161c155-a751-4207-a686-6154fb8949a9-kube-api-access-x6fv8" (OuterVolumeSpecName: "kube-api-access-x6fv8") pod "f161c155-a751-4207-a686-6154fb8949a9" (UID: "f161c155-a751-4207-a686-6154fb8949a9"). InnerVolumeSpecName "kube-api-access-x6fv8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:11:04.450228 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:11:04.450133    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f161c155-a751-4207-a686-6154fb8949a9-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "f161c155-a751-4207-a686-6154fb8949a9" (UID: "f161c155-a751-4207-a686-6154fb8949a9"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:11:04.548834 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:11:04.548803    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x6fv8\" (UniqueName: \"kubernetes.io/projected/f161c155-a751-4207-a686-6154fb8949a9-kube-api-access-x6fv8\") on node \"ip-10-0-139-237.ec2.internal\" DevicePath \"\""
Sep 17 18:11:04.548834 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:11:04.548827    2567 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f161c155-a751-4207-a686-6154fb8949a9-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-237.ec2.internal\" DevicePath \"\""
Sep 17 18:11:04.548834 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:11:04.548838    2567 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f161c155-a751-4207-a686-6154fb8949a9-proxy-tls\") on node \"ip-10-0-139-237.ec2.internal\" DevicePath \"\""
Sep 17 18:11:04.843417 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:11:04.843381    2567 generic.go:356] "Generic (PLEG): container finished" podID="f161c155-a751-4207-a686-6154fb8949a9" containerID="e63913250ccc577d9238c3f4d7ad26b296c597759a6968a164fec61a9f3bb50e" exitCode=0
Sep 17 18:11:04.843880 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:11:04.843430    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-9d702-predictor-5967c4d597-nkqp8"
Sep 17 18:11:04.843880 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:11:04.843468    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-9d702-predictor-5967c4d597-nkqp8" event={"ID":"f161c155-a751-4207-a686-6154fb8949a9","Type":"ContainerDied","Data":"e63913250ccc577d9238c3f4d7ad26b296c597759a6968a164fec61a9f3bb50e"}
Sep 17 18:11:04.843880 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:11:04.843549    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-9d702-predictor-5967c4d597-nkqp8" event={"ID":"f161c155-a751-4207-a686-6154fb8949a9","Type":"ContainerDied","Data":"6b44999d3af0a12ef9f678b909ead42cda1afb6f1aadd885312765da5be02192"}
Sep 17 18:11:04.843880 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:11:04.843570    2567 scope.go:117] "RemoveContainer" containerID="e91634bff0c24472ab73401dd9d8909baf410768c552e1665de1269316314a01"
Sep 17 18:11:04.852361 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:11:04.852343    2567 scope.go:117] "RemoveContainer" containerID="e63913250ccc577d9238c3f4d7ad26b296c597759a6968a164fec61a9f3bb50e"
Sep 17 18:11:04.860059 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:11:04.860040    2567 scope.go:117] "RemoveContainer" containerID="e91634bff0c24472ab73401dd9d8909baf410768c552e1665de1269316314a01"
Sep 17 18:11:04.860380 ip-10-0-139-237 kubenswrapper[2567]: E0917 18:11:04.860360    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e91634bff0c24472ab73401dd9d8909baf410768c552e1665de1269316314a01\": container with ID starting with e91634bff0c24472ab73401dd9d8909baf410768c552e1665de1269316314a01 not found: ID does not exist" containerID="e91634bff0c24472ab73401dd9d8909baf410768c552e1665de1269316314a01"
Sep 17 18:11:04.860440 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:11:04.860388    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e91634bff0c24472ab73401dd9d8909baf410768c552e1665de1269316314a01"} err="failed to get container status \"e91634bff0c24472ab73401dd9d8909baf410768c552e1665de1269316314a01\": rpc error: code = NotFound desc = could not find container \"e91634bff0c24472ab73401dd9d8909baf410768c552e1665de1269316314a01\": container with ID starting with e91634bff0c24472ab73401dd9d8909baf410768c552e1665de1269316314a01 not found: ID does not exist"
Sep 17 18:11:04.860440 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:11:04.860410    2567 scope.go:117] "RemoveContainer" containerID="e63913250ccc577d9238c3f4d7ad26b296c597759a6968a164fec61a9f3bb50e"
Sep 17 18:11:04.860720 ip-10-0-139-237 kubenswrapper[2567]: E0917 18:11:04.860701    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e63913250ccc577d9238c3f4d7ad26b296c597759a6968a164fec61a9f3bb50e\": container with ID starting with e63913250ccc577d9238c3f4d7ad26b296c597759a6968a164fec61a9f3bb50e not found: ID does not exist" containerID="e63913250ccc577d9238c3f4d7ad26b296c597759a6968a164fec61a9f3bb50e"
Sep 17 18:11:04.860784 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:11:04.860724    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e63913250ccc577d9238c3f4d7ad26b296c597759a6968a164fec61a9f3bb50e"} err="failed to get container status \"e63913250ccc577d9238c3f4d7ad26b296c597759a6968a164fec61a9f3bb50e\": rpc error: code = NotFound desc = could not find container \"e63913250ccc577d9238c3f4d7ad26b296c597759a6968a164fec61a9f3bb50e\": container with ID starting with e63913250ccc577d9238c3f4d7ad26b296c597759a6968a164fec61a9f3bb50e not found: ID does not exist"
Sep 17 18:11:04.882273 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:11:04.882248    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-9d702-predictor-5967c4d597-nkqp8"]
Sep 17 18:11:04.887830 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:11:04.887808    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-9d702-predictor-5967c4d597-nkqp8"]
Sep 17 18:11:05.285710 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:11:05.285633    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f161c155-a751-4207-a686-6154fb8949a9" path="/var/lib/kubelet/pods/f161c155-a751-4207-a686-6154fb8949a9/volumes"
Sep 17 18:11:08.826153 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:11:08.826118    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-9d702-6849cf5859-m6bk5" podUID="e5beaf37-1f94-4f07-8e24-2c92390ec75c" containerName="ensemble-graph-9d702" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 18:11:08.841597 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:11:08.841571    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-de19c-predictor-d5c8fcc89-47k8v"
Sep 17 18:11:08.842002 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:11:08.841976    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-de19c-predictor-d5c8fcc89-47k8v" podUID="ce283b34-ac81-4d48-a2c1-a75f93b3958d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Sep 17 18:11:13.826275 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:11:13.826237    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-9d702-6849cf5859-m6bk5" podUID="e5beaf37-1f94-4f07-8e24-2c92390ec75c" containerName="ensemble-graph-9d702" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 18:11:13.826661 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:11:13.826353    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/ensemble-graph-9d702-6849cf5859-m6bk5"
Sep 17 18:11:18.825778 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:11:18.825743    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-9d702-6849cf5859-m6bk5" podUID="e5beaf37-1f94-4f07-8e24-2c92390ec75c" containerName="ensemble-graph-9d702" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 18:11:18.842671 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:11:18.842635    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-de19c-predictor-d5c8fcc89-47k8v" podUID="ce283b34-ac81-4d48-a2c1-a75f93b3958d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Sep 17 18:11:23.825750 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:11:23.825711    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-9d702-6849cf5859-m6bk5" podUID="e5beaf37-1f94-4f07-8e24-2c92390ec75c" containerName="ensemble-graph-9d702" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 18:11:28.826272 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:11:28.826233    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-9d702-6849cf5859-m6bk5" podUID="e5beaf37-1f94-4f07-8e24-2c92390ec75c" containerName="ensemble-graph-9d702" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 18:11:28.842490 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:11:28.842456    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-de19c-predictor-d5c8fcc89-47k8v" podUID="ce283b34-ac81-4d48-a2c1-a75f93b3958d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Sep 17 18:11:30.921183 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:11:30.921158    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-9d702-6849cf5859-m6bk5"
Sep 17 18:11:30.951452 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:11:30.951421    2567 generic.go:356] "Generic (PLEG): container finished" podID="e5beaf37-1f94-4f07-8e24-2c92390ec75c" containerID="371541d0a024a2041ba16c804dac22058f22c660e795ec8b106d6e97baf8bf4d" exitCode=0
Sep 17 18:11:30.951614 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:11:30.951461    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-9d702-6849cf5859-m6bk5" event={"ID":"e5beaf37-1f94-4f07-8e24-2c92390ec75c","Type":"ContainerDied","Data":"371541d0a024a2041ba16c804dac22058f22c660e795ec8b106d6e97baf8bf4d"}
Sep 17 18:11:30.951614 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:11:30.951464    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-9d702-6849cf5859-m6bk5"
Sep 17 18:11:30.951614 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:11:30.951505    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-9d702-6849cf5859-m6bk5" event={"ID":"e5beaf37-1f94-4f07-8e24-2c92390ec75c","Type":"ContainerDied","Data":"fed0ba5b0f8ce06d8cf91ac7507a60c0a54643247cbb57dca8b85660c7eb6e80"}
Sep 17 18:11:30.951614 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:11:30.951542    2567 scope.go:117] "RemoveContainer" containerID="371541d0a024a2041ba16c804dac22058f22c660e795ec8b106d6e97baf8bf4d"
Sep 17 18:11:30.959348 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:11:30.959326    2567 scope.go:117] "RemoveContainer" containerID="371541d0a024a2041ba16c804dac22058f22c660e795ec8b106d6e97baf8bf4d"
Sep 17 18:11:30.959632 ip-10-0-139-237 kubenswrapper[2567]: E0917 18:11:30.959615    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"371541d0a024a2041ba16c804dac22058f22c660e795ec8b106d6e97baf8bf4d\": container with ID starting with 371541d0a024a2041ba16c804dac22058f22c660e795ec8b106d6e97baf8bf4d not found: ID does not exist" containerID="371541d0a024a2041ba16c804dac22058f22c660e795ec8b106d6e97baf8bf4d"
Sep 17 18:11:30.959693 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:11:30.959640    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"371541d0a024a2041ba16c804dac22058f22c660e795ec8b106d6e97baf8bf4d"} err="failed to get container status \"371541d0a024a2041ba16c804dac22058f22c660e795ec8b106d6e97baf8bf4d\": rpc error: code = NotFound desc = could not find container \"371541d0a024a2041ba16c804dac22058f22c660e795ec8b106d6e97baf8bf4d\": container with ID starting with 371541d0a024a2041ba16c804dac22058f22c660e795ec8b106d6e97baf8bf4d not found: ID does not exist"
Sep 17 18:11:30.968937 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:11:30.968909    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e5beaf37-1f94-4f07-8e24-2c92390ec75c-proxy-tls\") pod \"e5beaf37-1f94-4f07-8e24-2c92390ec75c\" (UID: \"e5beaf37-1f94-4f07-8e24-2c92390ec75c\") "
Sep 17 18:11:30.969040 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:11:30.968958    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e5beaf37-1f94-4f07-8e24-2c92390ec75c-openshift-service-ca-bundle\") pod \"e5beaf37-1f94-4f07-8e24-2c92390ec75c\" (UID: \"e5beaf37-1f94-4f07-8e24-2c92390ec75c\") "
Sep 17 18:11:30.969317 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:11:30.969297    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5beaf37-1f94-4f07-8e24-2c92390ec75c-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "e5beaf37-1f94-4f07-8e24-2c92390ec75c" (UID: "e5beaf37-1f94-4f07-8e24-2c92390ec75c"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:11:30.970915 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:11:30.970888    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e5beaf37-1f94-4f07-8e24-2c92390ec75c-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "e5beaf37-1f94-4f07-8e24-2c92390ec75c" (UID: "e5beaf37-1f94-4f07-8e24-2c92390ec75c"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:11:31.069977 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:11:31.069889    2567 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e5beaf37-1f94-4f07-8e24-2c92390ec75c-proxy-tls\") on node \"ip-10-0-139-237.ec2.internal\" DevicePath \"\""
Sep 17 18:11:31.069977 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:11:31.069921    2567 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e5beaf37-1f94-4f07-8e24-2c92390ec75c-openshift-service-ca-bundle\") on node \"ip-10-0-139-237.ec2.internal\" DevicePath \"\""
Sep 17 18:11:31.274484 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:11:31.274452    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-9d702-6849cf5859-m6bk5"]
Sep 17 18:11:31.277830 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:11:31.277805    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-9d702-6849cf5859-m6bk5"]
Sep 17 18:11:31.286489 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:11:31.286467    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e5beaf37-1f94-4f07-8e24-2c92390ec75c" path="/var/lib/kubelet/pods/e5beaf37-1f94-4f07-8e24-2c92390ec75c/volumes"
Sep 17 18:11:38.842050 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:11:38.842009    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-de19c-predictor-d5c8fcc89-47k8v" podUID="ce283b34-ac81-4d48-a2c1-a75f93b3958d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Sep 17 18:11:48.843448 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:11:48.843413    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-de19c-predictor-d5c8fcc89-47k8v"
Sep 17 18:11:50.944552 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:11:50.944507    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-8eede-bdd8448d-m85hx"]
Sep 17 18:11:50.944974 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:11:50.944712    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/sequence-graph-8eede-bdd8448d-m85hx" podUID="a4197f5d-92b4-44b7-a820-e3d30f8882a4" containerName="sequence-graph-8eede" containerID="cri-o://6459bdc4048b6f2a7445938daaa85d262475dfb1e48687d9e827068d93ae4d0d" gracePeriod=30
Sep 17 18:11:50.976304 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:11:50.976275    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 18:11:51.088494 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:11:51.088463    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-8eede-predictor-5d489c9599-98jkr"]
Sep 17 18:11:51.088758 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:11:51.088730    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-8eede-predictor-5d489c9599-98jkr" podUID="73a12907-2862-4ed2-9a2b-ef500261948f" containerName="kserve-container" containerID="cri-o://1f2c131eb3a75fa2fb6cbd107a5fd523700ab864e0a84882d992d221302ab329" gracePeriod=30
Sep 17 18:11:51.088834 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:11:51.088766    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-8eede-predictor-5d489c9599-98jkr" podUID="73a12907-2862-4ed2-9a2b-ef500261948f" containerName="kube-rbac-proxy" containerID="cri-o://81498851ff8cb7eaafaf0a477592765ecf452c763ccd20ad51756226e3763480" gracePeriod=30
Sep 17 18:11:51.096478 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:11:51.096447    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 18:11:51.183714 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:11:51.183681    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 18:11:51.269317 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:11:51.269284    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 18:11:51.293165 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:11:51.293137    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-85bfa-predictor-5d4d479565-4dlks"]
Sep 17 18:11:51.293461 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:11:51.293449    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e5beaf37-1f94-4f07-8e24-2c92390ec75c" containerName="ensemble-graph-9d702"
Sep 17 18:11:51.293505 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:11:51.293463    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="e5beaf37-1f94-4f07-8e24-2c92390ec75c" containerName="ensemble-graph-9d702"
Sep 17 18:11:51.293505 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:11:51.293478    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f161c155-a751-4207-a686-6154fb8949a9" containerName="kube-rbac-proxy"
Sep 17 18:11:51.293505 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:11:51.293486    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="f161c155-a751-4207-a686-6154fb8949a9" containerName="kube-rbac-proxy"
Sep 17 18:11:51.293505 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:11:51.293499    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f161c155-a751-4207-a686-6154fb8949a9" containerName="kserve-container"
Sep 17 18:11:51.293505 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:11:51.293504    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="f161c155-a751-4207-a686-6154fb8949a9" containerName="kserve-container"
Sep 17 18:11:51.293687 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:11:51.293567    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="f161c155-a751-4207-a686-6154fb8949a9" containerName="kserve-container"
Sep 17 18:11:51.293687 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:11:51.293577    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="e5beaf37-1f94-4f07-8e24-2c92390ec75c" containerName="ensemble-graph-9d702"
Sep 17 18:11:51.293687 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:11:51.293583    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="f161c155-a751-4207-a686-6154fb8949a9" containerName="kube-rbac-proxy"
Sep 17 18:11:51.296606 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:11:51.296590    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-85bfa-predictor-5d4d479565-4dlks"
Sep 17 18:11:51.299041 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:11:51.299019    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-85bfa-predictor-serving-cert\""
Sep 17 18:11:51.299041 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:11:51.299038    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-85bfa-kube-rbac-proxy-sar-config\""
Sep 17 18:11:51.313344 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:11:51.313322    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-85bfa-predictor-5d4d479565-4dlks"]
Sep 17 18:11:51.325197 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:11:51.325176    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1219c9c7-e3bb-408a-99aa-7cc5583a5cdf-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-85bfa-predictor-5d4d479565-4dlks\" (UID: \"1219c9c7-e3bb-408a-99aa-7cc5583a5cdf\") " pod="kserve-ci-e2e-test/error-404-isvc-85bfa-predictor-5d4d479565-4dlks"
Sep 17 18:11:51.325314 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:11:51.325215    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ts78w\" (UniqueName: \"kubernetes.io/projected/1219c9c7-e3bb-408a-99aa-7cc5583a5cdf-kube-api-access-ts78w\") pod \"error-404-isvc-85bfa-predictor-5d4d479565-4dlks\" (UID: \"1219c9c7-e3bb-408a-99aa-7cc5583a5cdf\") " pod="kserve-ci-e2e-test/error-404-isvc-85bfa-predictor-5d4d479565-4dlks"
Sep 17 18:11:51.325314 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:11:51.325247    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1219c9c7-e3bb-408a-99aa-7cc5583a5cdf-proxy-tls\") pod \"error-404-isvc-85bfa-predictor-5d4d479565-4dlks\" (UID: \"1219c9c7-e3bb-408a-99aa-7cc5583a5cdf\") " pod="kserve-ci-e2e-test/error-404-isvc-85bfa-predictor-5d4d479565-4dlks"
Sep 17 18:11:51.369469 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:11:51.369441    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 18:11:51.425975 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:11:51.425950    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1219c9c7-e3bb-408a-99aa-7cc5583a5cdf-proxy-tls\") pod \"error-404-isvc-85bfa-predictor-5d4d479565-4dlks\" (UID: \"1219c9c7-e3bb-408a-99aa-7cc5583a5cdf\") " pod="kserve-ci-e2e-test/error-404-isvc-85bfa-predictor-5d4d479565-4dlks"
Sep 17 18:11:51.426111 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:11:51.426025    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1219c9c7-e3bb-408a-99aa-7cc5583a5cdf-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-85bfa-predictor-5d4d479565-4dlks\" (UID: \"1219c9c7-e3bb-408a-99aa-7cc5583a5cdf\") " pod="kserve-ci-e2e-test/error-404-isvc-85bfa-predictor-5d4d479565-4dlks"
Sep 17 18:11:51.426111 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:11:51.426065    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ts78w\" (UniqueName: \"kubernetes.io/projected/1219c9c7-e3bb-408a-99aa-7cc5583a5cdf-kube-api-access-ts78w\") pod \"error-404-isvc-85bfa-predictor-5d4d479565-4dlks\" (UID: \"1219c9c7-e3bb-408a-99aa-7cc5583a5cdf\") " pod="kserve-ci-e2e-test/error-404-isvc-85bfa-predictor-5d4d479565-4dlks"
Sep 17 18:11:51.426685 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:11:51.426664    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1219c9c7-e3bb-408a-99aa-7cc5583a5cdf-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-85bfa-predictor-5d4d479565-4dlks\" (UID: \"1219c9c7-e3bb-408a-99aa-7cc5583a5cdf\") " pod="kserve-ci-e2e-test/error-404-isvc-85bfa-predictor-5d4d479565-4dlks"
Sep 17 18:11:51.428314 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:11:51.428296    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1219c9c7-e3bb-408a-99aa-7cc5583a5cdf-proxy-tls\") pod \"error-404-isvc-85bfa-predictor-5d4d479565-4dlks\" (UID: \"1219c9c7-e3bb-408a-99aa-7cc5583a5cdf\") " pod="kserve-ci-e2e-test/error-404-isvc-85bfa-predictor-5d4d479565-4dlks"
Sep 17 18:11:51.435241 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:11:51.435213    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ts78w\" (UniqueName: \"kubernetes.io/projected/1219c9c7-e3bb-408a-99aa-7cc5583a5cdf-kube-api-access-ts78w\") pod \"error-404-isvc-85bfa-predictor-5d4d479565-4dlks\" (UID: \"1219c9c7-e3bb-408a-99aa-7cc5583a5cdf\") " pod="kserve-ci-e2e-test/error-404-isvc-85bfa-predictor-5d4d479565-4dlks"
Sep 17 18:11:51.470419 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:11:51.470354    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 18:11:51.606816 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:11:51.606777    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-85bfa-predictor-5d4d479565-4dlks"
Sep 17 18:11:51.880769 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:11:51.880692    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-85bfa-predictor-5d4d479565-4dlks"]
Sep 17 18:11:51.883023 ip-10-0-139-237 kubenswrapper[2567]: W0917 18:11:51.882993    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1219c9c7_e3bb_408a_99aa_7cc5583a5cdf.slice/crio-1c76682e992c03949b42b45d69f5c6da117fd5b589ca8b84e558fd1b5d69002e WatchSource:0}: Error finding container 1c76682e992c03949b42b45d69f5c6da117fd5b589ca8b84e558fd1b5d69002e: Status 404 returned error can't find the container with id 1c76682e992c03949b42b45d69f5c6da117fd5b589ca8b84e558fd1b5d69002e
Sep 17 18:11:52.044270 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:11:52.044238    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-85bfa-predictor-5d4d479565-4dlks" event={"ID":"1219c9c7-e3bb-408a-99aa-7cc5583a5cdf","Type":"ContainerStarted","Data":"81cf7aa2575afbccc42a0e353f7d7990cf7c4f6504c0aeb15f59af789144f783"}
Sep 17 18:11:52.044719 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:11:52.044311    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-85bfa-predictor-5d4d479565-4dlks"
Sep 17 18:11:52.044719 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:11:52.044419    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-85bfa-predictor-5d4d479565-4dlks"
Sep 17 18:11:52.044719 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:11:52.044472    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-85bfa-predictor-5d4d479565-4dlks" event={"ID":"1219c9c7-e3bb-408a-99aa-7cc5583a5cdf","Type":"ContainerStarted","Data":"0a648b015612347b5fd399cdea034a2c3f9324a8bc4a6c6580605ce8bec47f1c"}
Sep 17 18:11:52.044719 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:11:52.044506    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-85bfa-predictor-5d4d479565-4dlks" event={"ID":"1219c9c7-e3bb-408a-99aa-7cc5583a5cdf","Type":"ContainerStarted","Data":"1c76682e992c03949b42b45d69f5c6da117fd5b589ca8b84e558fd1b5d69002e"}
Sep 17 18:11:52.045609 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:11:52.045575    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-85bfa-predictor-5d4d479565-4dlks" podUID="1219c9c7-e3bb-408a-99aa-7cc5583a5cdf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Sep 17 18:11:52.046230 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:11:52.046211    2567 generic.go:356] "Generic (PLEG): container finished" podID="73a12907-2862-4ed2-9a2b-ef500261948f" containerID="81498851ff8cb7eaafaf0a477592765ecf452c763ccd20ad51756226e3763480" exitCode=2
Sep 17 18:11:52.046301 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:11:52.046283    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-8eede-predictor-5d489c9599-98jkr" event={"ID":"73a12907-2862-4ed2-9a2b-ef500261948f","Type":"ContainerDied","Data":"81498851ff8cb7eaafaf0a477592765ecf452c763ccd20ad51756226e3763480"}
Sep 17 18:11:52.064356 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:11:52.064315    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-85bfa-predictor-5d4d479565-4dlks" podStartSLOduration=1.064303322 podStartE2EDuration="1.064303322s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:11:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:11:52.063641365 +0000 UTC m=+1853.087246515" watchObservedRunningTime="2026-09-17 18:11:52.064303322 +0000 UTC m=+1853.087908536"
Sep 17 18:11:53.033900 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:11:53.033852    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-8eede-bdd8448d-m85hx" podUID="a4197f5d-92b4-44b7-a820-e3d30f8882a4" containerName="sequence-graph-8eede" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 18:11:53.051124 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:11:53.051080    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-85bfa-predictor-5d4d479565-4dlks" podUID="1219c9c7-e3bb-408a-99aa-7cc5583a5cdf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Sep 17 18:11:54.330473 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:11:54.330452    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-8eede-predictor-5d489c9599-98jkr"
Sep 17 18:11:54.352390 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:11:54.352350    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/73a12907-2862-4ed2-9a2b-ef500261948f-kube-rbac-proxy-sar-config\") pod \"73a12907-2862-4ed2-9a2b-ef500261948f\" (UID: \"73a12907-2862-4ed2-9a2b-ef500261948f\") "
Sep 17 18:11:54.352592 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:11:54.352403    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k2pwn\" (UniqueName: \"kubernetes.io/projected/73a12907-2862-4ed2-9a2b-ef500261948f-kube-api-access-k2pwn\") pod \"73a12907-2862-4ed2-9a2b-ef500261948f\" (UID: \"73a12907-2862-4ed2-9a2b-ef500261948f\") "
Sep 17 18:11:54.352592 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:11:54.352515    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/73a12907-2862-4ed2-9a2b-ef500261948f-proxy-tls\") pod \"73a12907-2862-4ed2-9a2b-ef500261948f\" (UID: \"73a12907-2862-4ed2-9a2b-ef500261948f\") "
Sep 17 18:11:54.352855 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:11:54.352823    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73a12907-2862-4ed2-9a2b-ef500261948f-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "73a12907-2862-4ed2-9a2b-ef500261948f" (UID: "73a12907-2862-4ed2-9a2b-ef500261948f"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:11:54.353137 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:11:54.353116    2567 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/73a12907-2862-4ed2-9a2b-ef500261948f-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-237.ec2.internal\" DevicePath \"\""
Sep 17 18:11:54.354717 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:11:54.354689    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/73a12907-2862-4ed2-9a2b-ef500261948f-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "73a12907-2862-4ed2-9a2b-ef500261948f" (UID: "73a12907-2862-4ed2-9a2b-ef500261948f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:11:54.354919 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:11:54.354900    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73a12907-2862-4ed2-9a2b-ef500261948f-kube-api-access-k2pwn" (OuterVolumeSpecName: "kube-api-access-k2pwn") pod "73a12907-2862-4ed2-9a2b-ef500261948f" (UID: "73a12907-2862-4ed2-9a2b-ef500261948f"). InnerVolumeSpecName "kube-api-access-k2pwn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:11:54.454519 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:11:54.454448    2567 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/73a12907-2862-4ed2-9a2b-ef500261948f-proxy-tls\") on node \"ip-10-0-139-237.ec2.internal\" DevicePath \"\""
Sep 17 18:11:54.454519 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:11:54.454485    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k2pwn\" (UniqueName: \"kubernetes.io/projected/73a12907-2862-4ed2-9a2b-ef500261948f-kube-api-access-k2pwn\") on node \"ip-10-0-139-237.ec2.internal\" DevicePath \"\""
Sep 17 18:11:55.061038 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:11:55.061004    2567 generic.go:356] "Generic (PLEG): container finished" podID="73a12907-2862-4ed2-9a2b-ef500261948f" containerID="1f2c131eb3a75fa2fb6cbd107a5fd523700ab864e0a84882d992d221302ab329" exitCode=0
Sep 17 18:11:55.061213 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:11:55.061058    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-8eede-predictor-5d489c9599-98jkr"
Sep 17 18:11:55.061213 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:11:55.061063    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-8eede-predictor-5d489c9599-98jkr" event={"ID":"73a12907-2862-4ed2-9a2b-ef500261948f","Type":"ContainerDied","Data":"1f2c131eb3a75fa2fb6cbd107a5fd523700ab864e0a84882d992d221302ab329"}
Sep 17 18:11:55.061213 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:11:55.061123    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-8eede-predictor-5d489c9599-98jkr" event={"ID":"73a12907-2862-4ed2-9a2b-ef500261948f","Type":"ContainerDied","Data":"bcc404ba333179e2492066e101036f2e415346112ffd0eb167ef5e05002c11f7"}
Sep 17 18:11:55.061213 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:11:55.061161    2567 scope.go:117] "RemoveContainer" containerID="81498851ff8cb7eaafaf0a477592765ecf452c763ccd20ad51756226e3763480"
Sep 17 18:11:55.070056 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:11:55.070039    2567 scope.go:117] "RemoveContainer" containerID="1f2c131eb3a75fa2fb6cbd107a5fd523700ab864e0a84882d992d221302ab329"
Sep 17 18:11:55.077013 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:11:55.076998    2567 scope.go:117] "RemoveContainer" containerID="81498851ff8cb7eaafaf0a477592765ecf452c763ccd20ad51756226e3763480"
Sep 17 18:11:55.077263 ip-10-0-139-237 kubenswrapper[2567]: E0917 18:11:55.077245    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"81498851ff8cb7eaafaf0a477592765ecf452c763ccd20ad51756226e3763480\": container with ID starting with 81498851ff8cb7eaafaf0a477592765ecf452c763ccd20ad51756226e3763480 not found: ID does not exist" containerID="81498851ff8cb7eaafaf0a477592765ecf452c763ccd20ad51756226e3763480"
Sep 17 18:11:55.077302 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:11:55.077271    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81498851ff8cb7eaafaf0a477592765ecf452c763ccd20ad51756226e3763480"} err="failed to get container status \"81498851ff8cb7eaafaf0a477592765ecf452c763ccd20ad51756226e3763480\": rpc error: code = NotFound desc = could not find container \"81498851ff8cb7eaafaf0a477592765ecf452c763ccd20ad51756226e3763480\": container with ID starting with 81498851ff8cb7eaafaf0a477592765ecf452c763ccd20ad51756226e3763480 not found: ID does not exist"
Sep 17 18:11:55.077302 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:11:55.077285    2567 scope.go:117] "RemoveContainer" containerID="1f2c131eb3a75fa2fb6cbd107a5fd523700ab864e0a84882d992d221302ab329"
Sep 17 18:11:55.077495 ip-10-0-139-237 kubenswrapper[2567]: E0917 18:11:55.077481    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f2c131eb3a75fa2fb6cbd107a5fd523700ab864e0a84882d992d221302ab329\": container with ID starting with 1f2c131eb3a75fa2fb6cbd107a5fd523700ab864e0a84882d992d221302ab329 not found: ID does not exist" containerID="1f2c131eb3a75fa2fb6cbd107a5fd523700ab864e0a84882d992d221302ab329"
Sep 17 18:11:55.077545 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:11:55.077500    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f2c131eb3a75fa2fb6cbd107a5fd523700ab864e0a84882d992d221302ab329"} err="failed to get container status \"1f2c131eb3a75fa2fb6cbd107a5fd523700ab864e0a84882d992d221302ab329\": rpc error: code = NotFound desc = could not find container \"1f2c131eb3a75fa2fb6cbd107a5fd523700ab864e0a84882d992d221302ab329\": container with ID starting with 1f2c131eb3a75fa2fb6cbd107a5fd523700ab864e0a84882d992d221302ab329 not found: ID does not exist"
Sep 17 18:11:55.083446 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:11:55.083421    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-8eede-predictor-5d489c9599-98jkr"]
Sep 17 18:11:55.087248 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:11:55.087226    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-8eede-predictor-5d489c9599-98jkr"]
Sep 17 18:11:55.286322 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:11:55.286290    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="73a12907-2862-4ed2-9a2b-ef500261948f" path="/var/lib/kubelet/pods/73a12907-2862-4ed2-9a2b-ef500261948f/volumes"
Sep 17 18:11:58.034113 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:11:58.034072    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-8eede-bdd8448d-m85hx" podUID="a4197f5d-92b4-44b7-a820-e3d30f8882a4" containerName="sequence-graph-8eede" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 18:11:58.056224 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:11:58.056202    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-85bfa-predictor-5d4d479565-4dlks"
Sep 17 18:11:58.056717 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:11:58.056686    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-85bfa-predictor-5d4d479565-4dlks" podUID="1219c9c7-e3bb-408a-99aa-7cc5583a5cdf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Sep 17 18:12:01.052075 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:12:01.052037    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/splitter-graph-de19c-847bdf9fb9-z5b2d"]
Sep 17 18:12:01.052444 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:12:01.052366    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="73a12907-2862-4ed2-9a2b-ef500261948f" containerName="kserve-container"
Sep 17 18:12:01.052444 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:12:01.052378    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="73a12907-2862-4ed2-9a2b-ef500261948f" containerName="kserve-container"
Sep 17 18:12:01.052444 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:12:01.052395    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="73a12907-2862-4ed2-9a2b-ef500261948f" containerName="kube-rbac-proxy"
Sep 17 18:12:01.052444 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:12:01.052401    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="73a12907-2862-4ed2-9a2b-ef500261948f" containerName="kube-rbac-proxy"
Sep 17 18:12:01.052589 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:12:01.052459    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="73a12907-2862-4ed2-9a2b-ef500261948f" containerName="kube-rbac-proxy"
Sep 17 18:12:01.052589 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:12:01.052468    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="73a12907-2862-4ed2-9a2b-ef500261948f" containerName="kserve-container"
Sep 17 18:12:01.056873 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:12:01.056856    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-de19c-847bdf9fb9-z5b2d"
Sep 17 18:12:01.060244 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:12:01.060216    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"splitter-graph-de19c-serving-cert\""
Sep 17 18:12:01.060244 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:12:01.060218    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"splitter-graph-de19c-kube-rbac-proxy-sar-config\""
Sep 17 18:12:01.069511 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:12:01.069487    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-de19c-847bdf9fb9-z5b2d"]
Sep 17 18:12:01.075335 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:12:01.075307    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 18:12:01.107275 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:12:01.107245    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/09c323ff-00a3-4cc8-963b-1030a490a9cd-openshift-service-ca-bundle\") pod \"splitter-graph-de19c-847bdf9fb9-z5b2d\" (UID: \"09c323ff-00a3-4cc8-963b-1030a490a9cd\") " pod="kserve-ci-e2e-test/splitter-graph-de19c-847bdf9fb9-z5b2d"
Sep 17 18:12:01.107388 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:12:01.107301    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/09c323ff-00a3-4cc8-963b-1030a490a9cd-proxy-tls\") pod \"splitter-graph-de19c-847bdf9fb9-z5b2d\" (UID: \"09c323ff-00a3-4cc8-963b-1030a490a9cd\") " pod="kserve-ci-e2e-test/splitter-graph-de19c-847bdf9fb9-z5b2d"
Sep 17 18:12:01.207466 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:12:01.207423    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 18:12:01.207698 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:12:01.207679    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/09c323ff-00a3-4cc8-963b-1030a490a9cd-proxy-tls\") pod \"splitter-graph-de19c-847bdf9fb9-z5b2d\" (UID: \"09c323ff-00a3-4cc8-963b-1030a490a9cd\") " pod="kserve-ci-e2e-test/splitter-graph-de19c-847bdf9fb9-z5b2d"
Sep 17 18:12:01.207807 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:12:01.207770    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/09c323ff-00a3-4cc8-963b-1030a490a9cd-openshift-service-ca-bundle\") pod \"splitter-graph-de19c-847bdf9fb9-z5b2d\" (UID: \"09c323ff-00a3-4cc8-963b-1030a490a9cd\") " pod="kserve-ci-e2e-test/splitter-graph-de19c-847bdf9fb9-z5b2d"
Sep 17 18:12:01.208381 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:12:01.208361    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/09c323ff-00a3-4cc8-963b-1030a490a9cd-openshift-service-ca-bundle\") pod \"splitter-graph-de19c-847bdf9fb9-z5b2d\" (UID: \"09c323ff-00a3-4cc8-963b-1030a490a9cd\") " pod="kserve-ci-e2e-test/splitter-graph-de19c-847bdf9fb9-z5b2d"
Sep 17 18:12:01.210355 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:12:01.210338    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"splitter-graph-de19c-serving-cert\""
Sep 17 18:12:01.220758 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:12:01.220737    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/09c323ff-00a3-4cc8-963b-1030a490a9cd-proxy-tls\") pod \"splitter-graph-de19c-847bdf9fb9-z5b2d\" (UID: \"09c323ff-00a3-4cc8-963b-1030a490a9cd\") " pod="kserve-ci-e2e-test/splitter-graph-de19c-847bdf9fb9-z5b2d"
Sep 17 18:12:01.367017 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:12:01.366936    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-de19c-847bdf9fb9-z5b2d"
Sep 17 18:12:01.651437 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:12:01.651411    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-de19c-847bdf9fb9-z5b2d"]
Sep 17 18:12:01.654585 ip-10-0-139-237 kubenswrapper[2567]: W0917 18:12:01.654557    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09c323ff_00a3_4cc8_963b_1030a490a9cd.slice/crio-3b35fcff18ddcfbf30b0be291839bb52ae5aa651189087a24b0c37c3e781028b WatchSource:0}: Error finding container 3b35fcff18ddcfbf30b0be291839bb52ae5aa651189087a24b0c37c3e781028b: Status 404 returned error can't find the container with id 3b35fcff18ddcfbf30b0be291839bb52ae5aa651189087a24b0c37c3e781028b
Sep 17 18:12:02.091758 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:12:02.091725    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-de19c-847bdf9fb9-z5b2d" event={"ID":"09c323ff-00a3-4cc8-963b-1030a490a9cd","Type":"ContainerStarted","Data":"bd76b6aeb07542a3d4ab674383a4bc79bec8b94395ed52d77c420b6835def474"}
Sep 17 18:12:02.092126 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:12:02.091804    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-de19c-847bdf9fb9-z5b2d" event={"ID":"09c323ff-00a3-4cc8-963b-1030a490a9cd","Type":"ContainerStarted","Data":"3b35fcff18ddcfbf30b0be291839bb52ae5aa651189087a24b0c37c3e781028b"}
Sep 17 18:12:02.092126 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:12:02.091871    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/splitter-graph-de19c-847bdf9fb9-z5b2d"
Sep 17 18:12:02.131556 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:12:02.131500    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/splitter-graph-de19c-847bdf9fb9-z5b2d" podStartSLOduration=1.131481711 podStartE2EDuration="1.131481711s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:12:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:12:02.130209415 +0000 UTC m=+1863.153814550" watchObservedRunningTime="2026-09-17 18:12:02.131481711 +0000 UTC m=+1863.155086847"
Sep 17 18:12:03.033947 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:12:03.033908    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-8eede-bdd8448d-m85hx" podUID="a4197f5d-92b4-44b7-a820-e3d30f8882a4" containerName="sequence-graph-8eede" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 18:12:03.034125 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:12:03.034030    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-8eede-bdd8448d-m85hx"
Sep 17 18:12:08.033206 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:12:08.033172    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-8eede-bdd8448d-m85hx" podUID="a4197f5d-92b4-44b7-a820-e3d30f8882a4" containerName="sequence-graph-8eede" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 18:12:08.056852 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:12:08.056814    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-85bfa-predictor-5d4d479565-4dlks" podUID="1219c9c7-e3bb-408a-99aa-7cc5583a5cdf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Sep 17 18:12:08.101403 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:12:08.101380    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/splitter-graph-de19c-847bdf9fb9-z5b2d"
Sep 17 18:12:11.118330 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:12:11.118295    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-de19c-847bdf9fb9-z5b2d"]
Sep 17 18:12:11.118773 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:12:11.118481    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/splitter-graph-de19c-847bdf9fb9-z5b2d" podUID="09c323ff-00a3-4cc8-963b-1030a490a9cd" containerName="splitter-graph-de19c" containerID="cri-o://bd76b6aeb07542a3d4ab674383a4bc79bec8b94395ed52d77c420b6835def474" gracePeriod=30
Sep 17 18:12:11.148607 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:12:11.148565    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 18:12:11.260427 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:12:11.260340    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 18:12:11.271580 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:12:11.271553    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-de19c-predictor-d5c8fcc89-47k8v"]
Sep 17 18:12:11.271806 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:12:11.271775    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-de19c-predictor-d5c8fcc89-47k8v" podUID="ce283b34-ac81-4d48-a2c1-a75f93b3958d" containerName="kserve-container" containerID="cri-o://e005040450edc6bf68e69c712b5fc5d0277b17855d03da448bf73f90afe75a94" gracePeriod=30
Sep 17 18:12:11.271933 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:12:11.271807    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-de19c-predictor-d5c8fcc89-47k8v" podUID="ce283b34-ac81-4d48-a2c1-a75f93b3958d" containerName="kube-rbac-proxy" containerID="cri-o://99c80879680f9790fc8f2aba67a2ccbdcc331f218f4961fe9396cf57ac612323" gracePeriod=30
Sep 17 18:12:11.349358 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:12:11.349328    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 18:12:11.416652 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:12:11.416619    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 18:12:11.446862 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:12:11.446835    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-e7d45-predictor-bfcffd599-ntx59"]
Sep 17 18:12:11.450141 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:12:11.450124    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-e7d45-predictor-bfcffd599-ntx59"
Sep 17 18:12:11.452840 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:12:11.452809    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-e7d45-predictor-serving-cert\""
Sep 17 18:12:11.452976 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:12:11.452818    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-e7d45-kube-rbac-proxy-sar-config\""
Sep 17 18:12:11.459705 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:12:11.459676    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-e7d45-predictor-bfcffd599-ntx59"]
Sep 17 18:12:11.516802 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:12:11.516779    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 18:12:11.595979 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:12:11.595942    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6fdf86e4-2b55-40a4-8ad2-385ee66aebc5-proxy-tls\") pod \"error-404-isvc-e7d45-predictor-bfcffd599-ntx59\" (UID: \"6fdf86e4-2b55-40a4-8ad2-385ee66aebc5\") " pod="kserve-ci-e2e-test/error-404-isvc-e7d45-predictor-bfcffd599-ntx59"
Sep 17 18:12:11.596173 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:12:11.595999    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qmf74\" (UniqueName: \"kubernetes.io/projected/6fdf86e4-2b55-40a4-8ad2-385ee66aebc5-kube-api-access-qmf74\") pod \"error-404-isvc-e7d45-predictor-bfcffd599-ntx59\" (UID: \"6fdf86e4-2b55-40a4-8ad2-385ee66aebc5\") " pod="kserve-ci-e2e-test/error-404-isvc-e7d45-predictor-bfcffd599-ntx59"
Sep 17 18:12:11.596173 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:12:11.596083    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6fdf86e4-2b55-40a4-8ad2-385ee66aebc5-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-e7d45-predictor-bfcffd599-ntx59\" (UID: \"6fdf86e4-2b55-40a4-8ad2-385ee66aebc5\") " pod="kserve-ci-e2e-test/error-404-isvc-e7d45-predictor-bfcffd599-ntx59"
Sep 17 18:12:11.616877 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:12:11.616843    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 18:12:11.696832 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:12:11.696744    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6fdf86e4-2b55-40a4-8ad2-385ee66aebc5-proxy-tls\") pod \"error-404-isvc-e7d45-predictor-bfcffd599-ntx59\" (UID: \"6fdf86e4-2b55-40a4-8ad2-385ee66aebc5\") " pod="kserve-ci-e2e-test/error-404-isvc-e7d45-predictor-bfcffd599-ntx59"
Sep 17 18:12:11.696832 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:12:11.696796    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qmf74\" (UniqueName: \"kubernetes.io/projected/6fdf86e4-2b55-40a4-8ad2-385ee66aebc5-kube-api-access-qmf74\") pod \"error-404-isvc-e7d45-predictor-bfcffd599-ntx59\" (UID: \"6fdf86e4-2b55-40a4-8ad2-385ee66aebc5\") " pod="kserve-ci-e2e-test/error-404-isvc-e7d45-predictor-bfcffd599-ntx59"
Sep 17 18:12:11.696832 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:12:11.696830    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6fdf86e4-2b55-40a4-8ad2-385ee66aebc5-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-e7d45-predictor-bfcffd599-ntx59\" (UID: \"6fdf86e4-2b55-40a4-8ad2-385ee66aebc5\") " pod="kserve-ci-e2e-test/error-404-isvc-e7d45-predictor-bfcffd599-ntx59"
Sep 17 18:12:11.697527 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:12:11.697501    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6fdf86e4-2b55-40a4-8ad2-385ee66aebc5-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-e7d45-predictor-bfcffd599-ntx59\" (UID: \"6fdf86e4-2b55-40a4-8ad2-385ee66aebc5\") " pod="kserve-ci-e2e-test/error-404-isvc-e7d45-predictor-bfcffd599-ntx59"
Sep 17 18:12:11.699120 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:12:11.699101    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6fdf86e4-2b55-40a4-8ad2-385ee66aebc5-proxy-tls\") pod \"error-404-isvc-e7d45-predictor-bfcffd599-ntx59\" (UID: \"6fdf86e4-2b55-40a4-8ad2-385ee66aebc5\") " pod="kserve-ci-e2e-test/error-404-isvc-e7d45-predictor-bfcffd599-ntx59"
Sep 17 18:12:11.704659 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:12:11.704628    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qmf74\" (UniqueName: \"kubernetes.io/projected/6fdf86e4-2b55-40a4-8ad2-385ee66aebc5-kube-api-access-qmf74\") pod \"error-404-isvc-e7d45-predictor-bfcffd599-ntx59\" (UID: \"6fdf86e4-2b55-40a4-8ad2-385ee66aebc5\") " pod="kserve-ci-e2e-test/error-404-isvc-e7d45-predictor-bfcffd599-ntx59"
Sep 17 18:12:11.716859 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:12:11.716824    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 18:12:11.762210 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:12:11.762188    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-e7d45-predictor-bfcffd599-ntx59"
Sep 17 18:12:12.038512 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:12:12.038488    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-e7d45-predictor-bfcffd599-ntx59"]
Sep 17 18:12:12.040467 ip-10-0-139-237 kubenswrapper[2567]: W0917 18:12:12.040440    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6fdf86e4_2b55_40a4_8ad2_385ee66aebc5.slice/crio-48f36ddf5a429e24a6771bf5b14f8d5f45d298f7f25f9f7ab71f33a5e4009e82 WatchSource:0}: Error finding container 48f36ddf5a429e24a6771bf5b14f8d5f45d298f7f25f9f7ab71f33a5e4009e82: Status 404 returned error can't find the container with id 48f36ddf5a429e24a6771bf5b14f8d5f45d298f7f25f9f7ab71f33a5e4009e82
Sep 17 18:12:12.134096 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:12:12.134062    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-e7d45-predictor-bfcffd599-ntx59" event={"ID":"6fdf86e4-2b55-40a4-8ad2-385ee66aebc5","Type":"ContainerStarted","Data":"a9120d85c96ac88a77ad7e8ba5edad1b311f474a9c9cbeeb730841c328c5b1b4"}
Sep 17 18:12:12.134482 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:12:12.134121    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-e7d45-predictor-bfcffd599-ntx59" event={"ID":"6fdf86e4-2b55-40a4-8ad2-385ee66aebc5","Type":"ContainerStarted","Data":"48f36ddf5a429e24a6771bf5b14f8d5f45d298f7f25f9f7ab71f33a5e4009e82"}
Sep 17 18:12:12.135829 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:12:12.135806    2567 generic.go:356] "Generic (PLEG): container finished" podID="ce283b34-ac81-4d48-a2c1-a75f93b3958d" containerID="99c80879680f9790fc8f2aba67a2ccbdcc331f218f4961fe9396cf57ac612323" exitCode=2
Sep 17 18:12:12.135940 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:12:12.135844    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-de19c-predictor-d5c8fcc89-47k8v" event={"ID":"ce283b34-ac81-4d48-a2c1-a75f93b3958d","Type":"ContainerDied","Data":"99c80879680f9790fc8f2aba67a2ccbdcc331f218f4961fe9396cf57ac612323"}
Sep 17 18:12:13.033550 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:12:13.033499    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-8eede-bdd8448d-m85hx" podUID="a4197f5d-92b4-44b7-a820-e3d30f8882a4" containerName="sequence-graph-8eede" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 18:12:13.100191 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:12:13.100151    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-de19c-847bdf9fb9-z5b2d" podUID="09c323ff-00a3-4cc8-963b-1030a490a9cd" containerName="splitter-graph-de19c" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 18:12:13.141336 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:12:13.141305    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-e7d45-predictor-bfcffd599-ntx59" event={"ID":"6fdf86e4-2b55-40a4-8ad2-385ee66aebc5","Type":"ContainerStarted","Data":"04eb1c1acf3f9adfef17101e3363cf521b9225536676680da1705c2435d03d2d"}
Sep 17 18:12:13.141738 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:12:13.141463    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-e7d45-predictor-bfcffd599-ntx59"
Sep 17 18:12:13.141738 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:12:13.141608    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-e7d45-predictor-bfcffd599-ntx59"
Sep 17 18:12:13.142906 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:12:13.142884    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-e7d45-predictor-bfcffd599-ntx59" podUID="6fdf86e4-2b55-40a4-8ad2-385ee66aebc5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Sep 17 18:12:13.161320 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:12:13.161276    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-e7d45-predictor-bfcffd599-ntx59" podStartSLOduration=2.161262255 podStartE2EDuration="2.161262255s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:12:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:12:13.159597074 +0000 UTC m=+1874.183202243" watchObservedRunningTime="2026-09-17 18:12:13.161262255 +0000 UTC m=+1874.184867390"
Sep 17 18:12:13.838043 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:12:13.838001    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-de19c-predictor-d5c8fcc89-47k8v" podUID="ce283b34-ac81-4d48-a2c1-a75f93b3958d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.40:8643/healthz\": dial tcp 10.134.0.40:8643: connect: connection refused"
Sep 17 18:12:14.145899 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:12:14.145804    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-e7d45-predictor-bfcffd599-ntx59" podUID="6fdf86e4-2b55-40a4-8ad2-385ee66aebc5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Sep 17 18:12:14.516652 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:12:14.516629    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-de19c-predictor-d5c8fcc89-47k8v"
Sep 17 18:12:14.620940 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:12:14.620913    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ce283b34-ac81-4d48-a2c1-a75f93b3958d-proxy-tls\") pod \"ce283b34-ac81-4d48-a2c1-a75f93b3958d\" (UID: \"ce283b34-ac81-4d48-a2c1-a75f93b3958d\") "
Sep 17 18:12:14.621093 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:12:14.621004    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jsxfj\" (UniqueName: \"kubernetes.io/projected/ce283b34-ac81-4d48-a2c1-a75f93b3958d-kube-api-access-jsxfj\") pod \"ce283b34-ac81-4d48-a2c1-a75f93b3958d\" (UID: \"ce283b34-ac81-4d48-a2c1-a75f93b3958d\") "
Sep 17 18:12:14.621093 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:12:14.621053    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ce283b34-ac81-4d48-a2c1-a75f93b3958d-kube-rbac-proxy-sar-config\") pod \"ce283b34-ac81-4d48-a2c1-a75f93b3958d\" (UID: \"ce283b34-ac81-4d48-a2c1-a75f93b3958d\") "
Sep 17 18:12:14.621413 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:12:14.621389    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce283b34-ac81-4d48-a2c1-a75f93b3958d-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "ce283b34-ac81-4d48-a2c1-a75f93b3958d" (UID: "ce283b34-ac81-4d48-a2c1-a75f93b3958d"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:12:14.623061 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:12:14.623033    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ce283b34-ac81-4d48-a2c1-a75f93b3958d-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "ce283b34-ac81-4d48-a2c1-a75f93b3958d" (UID: "ce283b34-ac81-4d48-a2c1-a75f93b3958d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:12:14.623171 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:12:14.623100    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce283b34-ac81-4d48-a2c1-a75f93b3958d-kube-api-access-jsxfj" (OuterVolumeSpecName: "kube-api-access-jsxfj") pod "ce283b34-ac81-4d48-a2c1-a75f93b3958d" (UID: "ce283b34-ac81-4d48-a2c1-a75f93b3958d"). InnerVolumeSpecName "kube-api-access-jsxfj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:12:14.722009 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:12:14.721908    2567 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ce283b34-ac81-4d48-a2c1-a75f93b3958d-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-237.ec2.internal\" DevicePath \"\""
Sep 17 18:12:14.722009 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:12:14.721943    2567 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ce283b34-ac81-4d48-a2c1-a75f93b3958d-proxy-tls\") on node \"ip-10-0-139-237.ec2.internal\" DevicePath \"\""
Sep 17 18:12:14.722009 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:12:14.721957    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jsxfj\" (UniqueName: \"kubernetes.io/projected/ce283b34-ac81-4d48-a2c1-a75f93b3958d-kube-api-access-jsxfj\") on node \"ip-10-0-139-237.ec2.internal\" DevicePath \"\""
Sep 17 18:12:15.151155 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:12:15.151116    2567 generic.go:356] "Generic (PLEG): container finished" podID="ce283b34-ac81-4d48-a2c1-a75f93b3958d" containerID="e005040450edc6bf68e69c712b5fc5d0277b17855d03da448bf73f90afe75a94" exitCode=0
Sep 17 18:12:15.151655 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:12:15.151171    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-de19c-predictor-d5c8fcc89-47k8v"
Sep 17 18:12:15.151655 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:12:15.151202    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-de19c-predictor-d5c8fcc89-47k8v" event={"ID":"ce283b34-ac81-4d48-a2c1-a75f93b3958d","Type":"ContainerDied","Data":"e005040450edc6bf68e69c712b5fc5d0277b17855d03da448bf73f90afe75a94"}
Sep 17 18:12:15.151655 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:12:15.151302    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-de19c-predictor-d5c8fcc89-47k8v" event={"ID":"ce283b34-ac81-4d48-a2c1-a75f93b3958d","Type":"ContainerDied","Data":"c47cfeeea27101d294b438ecc70a34c080b3a209bd5dc0fef588ea26ccc3b862"}
Sep 17 18:12:15.151655 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:12:15.151341    2567 scope.go:117] "RemoveContainer" containerID="99c80879680f9790fc8f2aba67a2ccbdcc331f218f4961fe9396cf57ac612323"
Sep 17 18:12:15.160643 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:12:15.160624    2567 scope.go:117] "RemoveContainer" containerID="e005040450edc6bf68e69c712b5fc5d0277b17855d03da448bf73f90afe75a94"
Sep 17 18:12:15.168215 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:12:15.168195    2567 scope.go:117] "RemoveContainer" containerID="99c80879680f9790fc8f2aba67a2ccbdcc331f218f4961fe9396cf57ac612323"
Sep 17 18:12:15.168458 ip-10-0-139-237 kubenswrapper[2567]: E0917 18:12:15.168441    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"99c80879680f9790fc8f2aba67a2ccbdcc331f218f4961fe9396cf57ac612323\": container with ID starting with 99c80879680f9790fc8f2aba67a2ccbdcc331f218f4961fe9396cf57ac612323 not found: ID does not exist" containerID="99c80879680f9790fc8f2aba67a2ccbdcc331f218f4961fe9396cf57ac612323"
Sep 17 18:12:15.168506 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:12:15.168466    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"99c80879680f9790fc8f2aba67a2ccbdcc331f218f4961fe9396cf57ac612323"} err="failed to get container status \"99c80879680f9790fc8f2aba67a2ccbdcc331f218f4961fe9396cf57ac612323\": rpc error: code = NotFound desc = could not find container \"99c80879680f9790fc8f2aba67a2ccbdcc331f218f4961fe9396cf57ac612323\": container with ID starting with 99c80879680f9790fc8f2aba67a2ccbdcc331f218f4961fe9396cf57ac612323 not found: ID does not exist"
Sep 17 18:12:15.168506 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:12:15.168482    2567 scope.go:117] "RemoveContainer" containerID="e005040450edc6bf68e69c712b5fc5d0277b17855d03da448bf73f90afe75a94"
Sep 17 18:12:15.168745 ip-10-0-139-237 kubenswrapper[2567]: E0917 18:12:15.168729    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e005040450edc6bf68e69c712b5fc5d0277b17855d03da448bf73f90afe75a94\": container with ID starting with e005040450edc6bf68e69c712b5fc5d0277b17855d03da448bf73f90afe75a94 not found: ID does not exist" containerID="e005040450edc6bf68e69c712b5fc5d0277b17855d03da448bf73f90afe75a94"
Sep 17 18:12:15.168790 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:12:15.168747    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e005040450edc6bf68e69c712b5fc5d0277b17855d03da448bf73f90afe75a94"} err="failed to get container status \"e005040450edc6bf68e69c712b5fc5d0277b17855d03da448bf73f90afe75a94\": rpc error: code = NotFound desc = could not find container \"e005040450edc6bf68e69c712b5fc5d0277b17855d03da448bf73f90afe75a94\": container with ID starting with e005040450edc6bf68e69c712b5fc5d0277b17855d03da448bf73f90afe75a94 not found: ID does not exist"
Sep 17 18:12:15.174759 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:12:15.174738    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-de19c-predictor-d5c8fcc89-47k8v"]
Sep 17 18:12:15.178195 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:12:15.178175    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-de19c-predictor-d5c8fcc89-47k8v"]
Sep 17 18:12:15.286490 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:12:15.286458    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ce283b34-ac81-4d48-a2c1-a75f93b3958d" path="/var/lib/kubelet/pods/ce283b34-ac81-4d48-a2c1-a75f93b3958d/volumes"
Sep 17 18:12:18.034134 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:12:18.034094    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-8eede-bdd8448d-m85hx" podUID="a4197f5d-92b4-44b7-a820-e3d30f8882a4" containerName="sequence-graph-8eede" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 18:12:18.056759 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:12:18.056720    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-85bfa-predictor-5d4d479565-4dlks" podUID="1219c9c7-e3bb-408a-99aa-7cc5583a5cdf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Sep 17 18:12:18.099895 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:12:18.099866    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-de19c-847bdf9fb9-z5b2d" podUID="09c323ff-00a3-4cc8-963b-1030a490a9cd" containerName="splitter-graph-de19c" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 18:12:19.150917 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:12:19.150878    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-e7d45-predictor-bfcffd599-ntx59"
Sep 17 18:12:19.151461 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:12:19.151431    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-e7d45-predictor-bfcffd599-ntx59" podUID="6fdf86e4-2b55-40a4-8ad2-385ee66aebc5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Sep 17 18:12:21.089295 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:12:21.089269    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-8eede-bdd8448d-m85hx"
Sep 17 18:12:21.177627 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:12:21.177592    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a4197f5d-92b4-44b7-a820-e3d30f8882a4-openshift-service-ca-bundle\") pod \"a4197f5d-92b4-44b7-a820-e3d30f8882a4\" (UID: \"a4197f5d-92b4-44b7-a820-e3d30f8882a4\") "
Sep 17 18:12:21.177790 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:12:21.177656    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a4197f5d-92b4-44b7-a820-e3d30f8882a4-proxy-tls\") pod \"a4197f5d-92b4-44b7-a820-e3d30f8882a4\" (UID: \"a4197f5d-92b4-44b7-a820-e3d30f8882a4\") "
Sep 17 18:12:21.177930 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:12:21.177906    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4197f5d-92b4-44b7-a820-e3d30f8882a4-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "a4197f5d-92b4-44b7-a820-e3d30f8882a4" (UID: "a4197f5d-92b4-44b7-a820-e3d30f8882a4"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:12:21.180088 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:12:21.180064    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a4197f5d-92b4-44b7-a820-e3d30f8882a4-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a4197f5d-92b4-44b7-a820-e3d30f8882a4" (UID: "a4197f5d-92b4-44b7-a820-e3d30f8882a4"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:12:21.183304 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:12:21.183281    2567 generic.go:356] "Generic (PLEG): container finished" podID="a4197f5d-92b4-44b7-a820-e3d30f8882a4" containerID="6459bdc4048b6f2a7445938daaa85d262475dfb1e48687d9e827068d93ae4d0d" exitCode=0
Sep 17 18:12:21.183397 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:12:21.183321    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-8eede-bdd8448d-m85hx" event={"ID":"a4197f5d-92b4-44b7-a820-e3d30f8882a4","Type":"ContainerDied","Data":"6459bdc4048b6f2a7445938daaa85d262475dfb1e48687d9e827068d93ae4d0d"}
Sep 17 18:12:21.183397 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:12:21.183324    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-8eede-bdd8448d-m85hx"
Sep 17 18:12:21.183397 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:12:21.183358    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-8eede-bdd8448d-m85hx" event={"ID":"a4197f5d-92b4-44b7-a820-e3d30f8882a4","Type":"ContainerDied","Data":"152d090559b46a13c0a988e96ae31fa25fb6cbca23582dad7b717726885fe589"}
Sep 17 18:12:21.183397 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:12:21.183381    2567 scope.go:117] "RemoveContainer" containerID="6459bdc4048b6f2a7445938daaa85d262475dfb1e48687d9e827068d93ae4d0d"
Sep 17 18:12:21.191553 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:12:21.191521    2567 scope.go:117] "RemoveContainer" containerID="6459bdc4048b6f2a7445938daaa85d262475dfb1e48687d9e827068d93ae4d0d"
Sep 17 18:12:21.191832 ip-10-0-139-237 kubenswrapper[2567]: E0917 18:12:21.191813    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6459bdc4048b6f2a7445938daaa85d262475dfb1e48687d9e827068d93ae4d0d\": container with ID starting with 6459bdc4048b6f2a7445938daaa85d262475dfb1e48687d9e827068d93ae4d0d not found: ID does not exist" containerID="6459bdc4048b6f2a7445938daaa85d262475dfb1e48687d9e827068d93ae4d0d"
Sep 17 18:12:21.191900 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:12:21.191844    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6459bdc4048b6f2a7445938daaa85d262475dfb1e48687d9e827068d93ae4d0d"} err="failed to get container status \"6459bdc4048b6f2a7445938daaa85d262475dfb1e48687d9e827068d93ae4d0d\": rpc error: code = NotFound desc = could not find container \"6459bdc4048b6f2a7445938daaa85d262475dfb1e48687d9e827068d93ae4d0d\": container with ID starting with 6459bdc4048b6f2a7445938daaa85d262475dfb1e48687d9e827068d93ae4d0d not found: ID does not exist"
Sep 17 18:12:21.204167 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:12:21.204117    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-8eede-bdd8448d-m85hx"]
Sep 17 18:12:21.212008 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:12:21.211984    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-8eede-bdd8448d-m85hx"]
Sep 17 18:12:21.278817 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:12:21.278776    2567 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a4197f5d-92b4-44b7-a820-e3d30f8882a4-openshift-service-ca-bundle\") on node \"ip-10-0-139-237.ec2.internal\" DevicePath \"\""
Sep 17 18:12:21.278817 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:12:21.278812    2567 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a4197f5d-92b4-44b7-a820-e3d30f8882a4-proxy-tls\") on node \"ip-10-0-139-237.ec2.internal\" DevicePath \"\""
Sep 17 18:12:21.290584 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:12:21.290552    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a4197f5d-92b4-44b7-a820-e3d30f8882a4" path="/var/lib/kubelet/pods/a4197f5d-92b4-44b7-a820-e3d30f8882a4/volumes"
Sep 17 18:12:23.099672 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:12:23.099629    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-de19c-847bdf9fb9-z5b2d" podUID="09c323ff-00a3-4cc8-963b-1030a490a9cd" containerName="splitter-graph-de19c" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 18:12:23.100065 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:12:23.099735    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/splitter-graph-de19c-847bdf9fb9-z5b2d"
Sep 17 18:12:28.057237 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:12:28.057191    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-85bfa-predictor-5d4d479565-4dlks" podUID="1219c9c7-e3bb-408a-99aa-7cc5583a5cdf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Sep 17 18:12:28.099893 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:12:28.099857    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-de19c-847bdf9fb9-z5b2d" podUID="09c323ff-00a3-4cc8-963b-1030a490a9cd" containerName="splitter-graph-de19c" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 18:12:29.151726 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:12:29.151679    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-e7d45-predictor-bfcffd599-ntx59" podUID="6fdf86e4-2b55-40a4-8ad2-385ee66aebc5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Sep 17 18:12:33.099769 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:12:33.099731    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-de19c-847bdf9fb9-z5b2d" podUID="09c323ff-00a3-4cc8-963b-1030a490a9cd" containerName="splitter-graph-de19c" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 18:12:38.057142 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:12:38.057110    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-85bfa-predictor-5d4d479565-4dlks"
Sep 17 18:12:38.099767 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:12:38.099728    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-de19c-847bdf9fb9-z5b2d" podUID="09c323ff-00a3-4cc8-963b-1030a490a9cd" containerName="splitter-graph-de19c" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 18:12:39.151810 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:12:39.151774    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-e7d45-predictor-bfcffd599-ntx59" podUID="6fdf86e4-2b55-40a4-8ad2-385ee66aebc5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Sep 17 18:12:41.262216 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:12:41.262192    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-de19c-847bdf9fb9-z5b2d"
Sep 17 18:12:41.267767 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:12:41.267742    2567 generic.go:356] "Generic (PLEG): container finished" podID="09c323ff-00a3-4cc8-963b-1030a490a9cd" containerID="bd76b6aeb07542a3d4ab674383a4bc79bec8b94395ed52d77c420b6835def474" exitCode=0
Sep 17 18:12:41.267875 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:12:41.267775    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-de19c-847bdf9fb9-z5b2d"
Sep 17 18:12:41.267875 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:12:41.267780    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-de19c-847bdf9fb9-z5b2d" event={"ID":"09c323ff-00a3-4cc8-963b-1030a490a9cd","Type":"ContainerDied","Data":"bd76b6aeb07542a3d4ab674383a4bc79bec8b94395ed52d77c420b6835def474"}
Sep 17 18:12:41.267875 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:12:41.267820    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-de19c-847bdf9fb9-z5b2d" event={"ID":"09c323ff-00a3-4cc8-963b-1030a490a9cd","Type":"ContainerDied","Data":"3b35fcff18ddcfbf30b0be291839bb52ae5aa651189087a24b0c37c3e781028b"}
Sep 17 18:12:41.267875 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:12:41.267854    2567 scope.go:117] "RemoveContainer" containerID="bd76b6aeb07542a3d4ab674383a4bc79bec8b94395ed52d77c420b6835def474"
Sep 17 18:12:41.275375 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:12:41.275353    2567 scope.go:117] "RemoveContainer" containerID="bd76b6aeb07542a3d4ab674383a4bc79bec8b94395ed52d77c420b6835def474"
Sep 17 18:12:41.275651 ip-10-0-139-237 kubenswrapper[2567]: E0917 18:12:41.275635    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd76b6aeb07542a3d4ab674383a4bc79bec8b94395ed52d77c420b6835def474\": container with ID starting with bd76b6aeb07542a3d4ab674383a4bc79bec8b94395ed52d77c420b6835def474 not found: ID does not exist" containerID="bd76b6aeb07542a3d4ab674383a4bc79bec8b94395ed52d77c420b6835def474"
Sep 17 18:12:41.275728 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:12:41.275659    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd76b6aeb07542a3d4ab674383a4bc79bec8b94395ed52d77c420b6835def474"} err="failed to get container status \"bd76b6aeb07542a3d4ab674383a4bc79bec8b94395ed52d77c420b6835def474\": rpc error: code = NotFound desc = could not find container \"bd76b6aeb07542a3d4ab674383a4bc79bec8b94395ed52d77c420b6835def474\": container with ID starting with bd76b6aeb07542a3d4ab674383a4bc79bec8b94395ed52d77c420b6835def474 not found: ID does not exist"
Sep 17 18:12:41.352266 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:12:41.352235    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/09c323ff-00a3-4cc8-963b-1030a490a9cd-proxy-tls\") pod \"09c323ff-00a3-4cc8-963b-1030a490a9cd\" (UID: \"09c323ff-00a3-4cc8-963b-1030a490a9cd\") "
Sep 17 18:12:41.352460 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:12:41.352296    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/09c323ff-00a3-4cc8-963b-1030a490a9cd-openshift-service-ca-bundle\") pod \"09c323ff-00a3-4cc8-963b-1030a490a9cd\" (UID: \"09c323ff-00a3-4cc8-963b-1030a490a9cd\") "
Sep 17 18:12:41.352702 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:12:41.352678    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09c323ff-00a3-4cc8-963b-1030a490a9cd-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "09c323ff-00a3-4cc8-963b-1030a490a9cd" (UID: "09c323ff-00a3-4cc8-963b-1030a490a9cd"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:12:41.354252 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:12:41.354215    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/09c323ff-00a3-4cc8-963b-1030a490a9cd-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "09c323ff-00a3-4cc8-963b-1030a490a9cd" (UID: "09c323ff-00a3-4cc8-963b-1030a490a9cd"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:12:41.453807 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:12:41.453731    2567 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/09c323ff-00a3-4cc8-963b-1030a490a9cd-proxy-tls\") on node \"ip-10-0-139-237.ec2.internal\" DevicePath \"\""
Sep 17 18:12:41.453807 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:12:41.453767    2567 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/09c323ff-00a3-4cc8-963b-1030a490a9cd-openshift-service-ca-bundle\") on node \"ip-10-0-139-237.ec2.internal\" DevicePath \"\""
Sep 17 18:12:41.591768 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:12:41.591739    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-de19c-847bdf9fb9-z5b2d"]
Sep 17 18:12:41.595630 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:12:41.595606    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-de19c-847bdf9fb9-z5b2d"]
Sep 17 18:12:43.285794 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:12:43.285761    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="09c323ff-00a3-4cc8-963b-1030a490a9cd" path="/var/lib/kubelet/pods/09c323ff-00a3-4cc8-963b-1030a490a9cd/volumes"
Sep 17 18:12:49.152017 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:12:49.151967    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-e7d45-predictor-bfcffd599-ntx59" podUID="6fdf86e4-2b55-40a4-8ad2-385ee66aebc5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Sep 17 18:12:51.196495 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:12:51.196463    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/switch-graph-85bfa-7886b55cbc-fnw2h"]
Sep 17 18:12:51.196883 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:12:51.196867    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ce283b34-ac81-4d48-a2c1-a75f93b3958d" containerName="kserve-container"
Sep 17 18:12:51.196926 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:12:51.196884    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="ce283b34-ac81-4d48-a2c1-a75f93b3958d" containerName="kserve-container"
Sep 17 18:12:51.196926 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:12:51.196904    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a4197f5d-92b4-44b7-a820-e3d30f8882a4" containerName="sequence-graph-8eede"
Sep 17 18:12:51.196926 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:12:51.196909    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="a4197f5d-92b4-44b7-a820-e3d30f8882a4" containerName="sequence-graph-8eede"
Sep 17 18:12:51.196926 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:12:51.196917    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ce283b34-ac81-4d48-a2c1-a75f93b3958d" containerName="kube-rbac-proxy"
Sep 17 18:12:51.196926 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:12:51.196923    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="ce283b34-ac81-4d48-a2c1-a75f93b3958d" containerName="kube-rbac-proxy"
Sep 17 18:12:51.197080 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:12:51.196931    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="09c323ff-00a3-4cc8-963b-1030a490a9cd" containerName="splitter-graph-de19c"
Sep 17 18:12:51.197080 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:12:51.196936    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="09c323ff-00a3-4cc8-963b-1030a490a9cd" containerName="splitter-graph-de19c"
Sep 17 18:12:51.197080 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:12:51.196985    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="09c323ff-00a3-4cc8-963b-1030a490a9cd" containerName="splitter-graph-de19c"
Sep 17 18:12:51.197080 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:12:51.196992    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="a4197f5d-92b4-44b7-a820-e3d30f8882a4" containerName="sequence-graph-8eede"
Sep 17 18:12:51.197080 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:12:51.196997    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="ce283b34-ac81-4d48-a2c1-a75f93b3958d" containerName="kserve-container"
Sep 17 18:12:51.197080 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:12:51.197001    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="ce283b34-ac81-4d48-a2c1-a75f93b3958d" containerName="kube-rbac-proxy"
Sep 17 18:12:51.201465 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:12:51.201443    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-85bfa-7886b55cbc-fnw2h"
Sep 17 18:12:51.202516 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:12:51.202468    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 18:12:51.204158 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:12:51.204132    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-85bfa-kube-rbac-proxy-sar-config\""
Sep 17 18:12:51.204262 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:12:51.204248    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-85bfa-serving-cert\""
Sep 17 18:12:51.210266 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:12:51.210245    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-85bfa-7886b55cbc-fnw2h"]
Sep 17 18:12:51.236360 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:12:51.236338    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6d83c913-069b-43b9-af7a-b332934a9840-proxy-tls\") pod \"switch-graph-85bfa-7886b55cbc-fnw2h\" (UID: \"6d83c913-069b-43b9-af7a-b332934a9840\") " pod="kserve-ci-e2e-test/switch-graph-85bfa-7886b55cbc-fnw2h"
Sep 17 18:12:51.236466 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:12:51.236400    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6d83c913-069b-43b9-af7a-b332934a9840-openshift-service-ca-bundle\") pod \"switch-graph-85bfa-7886b55cbc-fnw2h\" (UID: \"6d83c913-069b-43b9-af7a-b332934a9840\") " pod="kserve-ci-e2e-test/switch-graph-85bfa-7886b55cbc-fnw2h"
Sep 17 18:12:51.337767 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:12:51.337727    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6d83c913-069b-43b9-af7a-b332934a9840-proxy-tls\") pod \"switch-graph-85bfa-7886b55cbc-fnw2h\" (UID: \"6d83c913-069b-43b9-af7a-b332934a9840\") " pod="kserve-ci-e2e-test/switch-graph-85bfa-7886b55cbc-fnw2h"
Sep 17 18:12:51.337936 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:12:51.337828    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6d83c913-069b-43b9-af7a-b332934a9840-openshift-service-ca-bundle\") pod \"switch-graph-85bfa-7886b55cbc-fnw2h\" (UID: \"6d83c913-069b-43b9-af7a-b332934a9840\") " pod="kserve-ci-e2e-test/switch-graph-85bfa-7886b55cbc-fnw2h"
Sep 17 18:12:51.337936 ip-10-0-139-237 kubenswrapper[2567]: E0917 18:12:51.337881    2567 secret.go:189] Couldn't get secret kserve-ci-e2e-test/switch-graph-85bfa-serving-cert: secret "switch-graph-85bfa-serving-cert" not found
Sep 17 18:12:51.338087 ip-10-0-139-237 kubenswrapper[2567]: E0917 18:12:51.337953    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6d83c913-069b-43b9-af7a-b332934a9840-proxy-tls podName:6d83c913-069b-43b9-af7a-b332934a9840 nodeName:}" failed. No retries permitted until 2026-09-17 18:12:51.837933374 +0000 UTC m=+1912.861538505 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/6d83c913-069b-43b9-af7a-b332934a9840-proxy-tls") pod "switch-graph-85bfa-7886b55cbc-fnw2h" (UID: "6d83c913-069b-43b9-af7a-b332934a9840") : secret "switch-graph-85bfa-serving-cert" not found
Sep 17 18:12:51.338437 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:12:51.338419    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6d83c913-069b-43b9-af7a-b332934a9840-openshift-service-ca-bundle\") pod \"switch-graph-85bfa-7886b55cbc-fnw2h\" (UID: \"6d83c913-069b-43b9-af7a-b332934a9840\") " pod="kserve-ci-e2e-test/switch-graph-85bfa-7886b55cbc-fnw2h"
Sep 17 18:12:51.597193 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:12:51.597157    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 18:12:51.842186 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:12:51.842153    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6d83c913-069b-43b9-af7a-b332934a9840-proxy-tls\") pod \"switch-graph-85bfa-7886b55cbc-fnw2h\" (UID: \"6d83c913-069b-43b9-af7a-b332934a9840\") " pod="kserve-ci-e2e-test/switch-graph-85bfa-7886b55cbc-fnw2h"
Sep 17 18:12:51.844524 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:12:51.844497    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6d83c913-069b-43b9-af7a-b332934a9840-proxy-tls\") pod \"switch-graph-85bfa-7886b55cbc-fnw2h\" (UID: \"6d83c913-069b-43b9-af7a-b332934a9840\") " pod="kserve-ci-e2e-test/switch-graph-85bfa-7886b55cbc-fnw2h"
Sep 17 18:12:52.112083 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:12:52.112046    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-85bfa-7886b55cbc-fnw2h"
Sep 17 18:12:52.383038 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:12:52.382965    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-85bfa-7886b55cbc-fnw2h"]
Sep 17 18:12:52.386321 ip-10-0-139-237 kubenswrapper[2567]: W0917 18:12:52.386293    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d83c913_069b_43b9_af7a_b332934a9840.slice/crio-86d85516b0dee6146e30260b2c75f1e5c5469937b69b6089bdc64762cc3cb3f5 WatchSource:0}: Error finding container 86d85516b0dee6146e30260b2c75f1e5c5469937b69b6089bdc64762cc3cb3f5: Status 404 returned error can't find the container with id 86d85516b0dee6146e30260b2c75f1e5c5469937b69b6089bdc64762cc3cb3f5
Sep 17 18:12:53.319973 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:12:53.319937    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-85bfa-7886b55cbc-fnw2h" event={"ID":"6d83c913-069b-43b9-af7a-b332934a9840","Type":"ContainerStarted","Data":"83475982114d8106c276bb05f8ec9548b3eec8f90b8cc821a58c84f602bb165f"}
Sep 17 18:12:53.320160 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:12:53.319993    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-85bfa-7886b55cbc-fnw2h" event={"ID":"6d83c913-069b-43b9-af7a-b332934a9840","Type":"ContainerStarted","Data":"86d85516b0dee6146e30260b2c75f1e5c5469937b69b6089bdc64762cc3cb3f5"}
Sep 17 18:12:53.320160 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:12:53.320021    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-85bfa-7886b55cbc-fnw2h"
Sep 17 18:12:53.337146 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:12:53.337096    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/switch-graph-85bfa-7886b55cbc-fnw2h" podStartSLOduration=2.337078911 podStartE2EDuration="2.337078911s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:12:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:12:53.336734033 +0000 UTC m=+1914.360339191" watchObservedRunningTime="2026-09-17 18:12:53.337078911 +0000 UTC m=+1914.360684047"
Sep 17 18:12:59.152680 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:12:59.152646    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-e7d45-predictor-bfcffd599-ntx59"
Sep 17 18:12:59.329633 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:12:59.329607    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/switch-graph-85bfa-7886b55cbc-fnw2h"
Sep 17 18:13:11.370933 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:13:11.370902    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/splitter-graph-e7d45-7d4894f44c-9lvpm"]
Sep 17 18:13:11.375251 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:13:11.375232    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-e7d45-7d4894f44c-9lvpm"
Sep 17 18:13:11.377625 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:13:11.377599    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"splitter-graph-e7d45-kube-rbac-proxy-sar-config\""
Sep 17 18:13:11.377625 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:13:11.377606    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"splitter-graph-e7d45-serving-cert\""
Sep 17 18:13:11.383861 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:13:11.383841    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-e7d45-7d4894f44c-9lvpm"]
Sep 17 18:13:11.384797 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:13:11.384761    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 18:13:11.492385 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:13:11.492347    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 18:13:11.512391 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:13:11.512369    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2e9f62b6-51ab-469c-8b94-3839312c1653-openshift-service-ca-bundle\") pod \"splitter-graph-e7d45-7d4894f44c-9lvpm\" (UID: \"2e9f62b6-51ab-469c-8b94-3839312c1653\") " pod="kserve-ci-e2e-test/splitter-graph-e7d45-7d4894f44c-9lvpm"
Sep 17 18:13:11.512488 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:13:11.512433    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2e9f62b6-51ab-469c-8b94-3839312c1653-proxy-tls\") pod \"splitter-graph-e7d45-7d4894f44c-9lvpm\" (UID: \"2e9f62b6-51ab-469c-8b94-3839312c1653\") " pod="kserve-ci-e2e-test/splitter-graph-e7d45-7d4894f44c-9lvpm"
Sep 17 18:13:11.613000 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:13:11.612968    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2e9f62b6-51ab-469c-8b94-3839312c1653-openshift-service-ca-bundle\") pod \"splitter-graph-e7d45-7d4894f44c-9lvpm\" (UID: \"2e9f62b6-51ab-469c-8b94-3839312c1653\") " pod="kserve-ci-e2e-test/splitter-graph-e7d45-7d4894f44c-9lvpm"
Sep 17 18:13:11.613173 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:13:11.613059    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2e9f62b6-51ab-469c-8b94-3839312c1653-proxy-tls\") pod \"splitter-graph-e7d45-7d4894f44c-9lvpm\" (UID: \"2e9f62b6-51ab-469c-8b94-3839312c1653\") " pod="kserve-ci-e2e-test/splitter-graph-e7d45-7d4894f44c-9lvpm"
Sep 17 18:13:11.613627 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:13:11.613606    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2e9f62b6-51ab-469c-8b94-3839312c1653-openshift-service-ca-bundle\") pod \"splitter-graph-e7d45-7d4894f44c-9lvpm\" (UID: \"2e9f62b6-51ab-469c-8b94-3839312c1653\") " pod="kserve-ci-e2e-test/splitter-graph-e7d45-7d4894f44c-9lvpm"
Sep 17 18:13:11.615498 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:13:11.615480    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2e9f62b6-51ab-469c-8b94-3839312c1653-proxy-tls\") pod \"splitter-graph-e7d45-7d4894f44c-9lvpm\" (UID: \"2e9f62b6-51ab-469c-8b94-3839312c1653\") " pod="kserve-ci-e2e-test/splitter-graph-e7d45-7d4894f44c-9lvpm"
Sep 17 18:13:11.686513 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:13:11.686450    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-e7d45-7d4894f44c-9lvpm"
Sep 17 18:13:11.957059 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:13:11.956975    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-e7d45-7d4894f44c-9lvpm"]
Sep 17 18:13:11.960892 ip-10-0-139-237 kubenswrapper[2567]: W0917 18:13:11.960866    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e9f62b6_51ab_469c_8b94_3839312c1653.slice/crio-823430a334ce67cec876f885b50278448f573955ef6b70659d9ac7127e205cfd WatchSource:0}: Error finding container 823430a334ce67cec876f885b50278448f573955ef6b70659d9ac7127e205cfd: Status 404 returned error can't find the container with id 823430a334ce67cec876f885b50278448f573955ef6b70659d9ac7127e205cfd
Sep 17 18:13:12.394170 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:13:12.394136    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-e7d45-7d4894f44c-9lvpm" event={"ID":"2e9f62b6-51ab-469c-8b94-3839312c1653","Type":"ContainerStarted","Data":"cbb6f39290aa63abcf54cfca98c7a6c609e91769d46d3fbe3d7a4e789f519304"}
Sep 17 18:13:12.394563 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:13:12.394186    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-e7d45-7d4894f44c-9lvpm" event={"ID":"2e9f62b6-51ab-469c-8b94-3839312c1653","Type":"ContainerStarted","Data":"823430a334ce67cec876f885b50278448f573955ef6b70659d9ac7127e205cfd"}
Sep 17 18:13:12.394563 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:13:12.394251    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/splitter-graph-e7d45-7d4894f44c-9lvpm"
Sep 17 18:13:12.411586 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:13:12.411542    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/splitter-graph-e7d45-7d4894f44c-9lvpm" podStartSLOduration=1.411513318 podStartE2EDuration="1.411513318s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:13:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:13:12.409910081 +0000 UTC m=+1933.433515224" watchObservedRunningTime="2026-09-17 18:13:12.411513318 +0000 UTC m=+1933.435118454"
Sep 17 18:13:18.403810 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:13:18.403781    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/splitter-graph-e7d45-7d4894f44c-9lvpm"
Sep 17 18:14:57.430469 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:14:57.430434    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 18:15:04.789430 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:15:04.789390    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 18:15:04.890031 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:15:04.889979    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 18:20:34.877318 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:20:34.877284    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 18:20:45.878189 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:20:45.878155    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 18:21:22.339682 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:21:22.339605    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 18:21:25.970786 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:21:25.970755    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-e7d45-7d4894f44c-9lvpm"]
Sep 17 18:21:25.971248 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:21:25.970965    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/splitter-graph-e7d45-7d4894f44c-9lvpm" podUID="2e9f62b6-51ab-469c-8b94-3839312c1653" containerName="splitter-graph-e7d45" containerID="cri-o://cbb6f39290aa63abcf54cfca98c7a6c609e91769d46d3fbe3d7a4e789f519304" gracePeriod=30
Sep 17 18:21:25.990849 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:21:25.990821    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 18:21:26.114692 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:21:26.114660    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 18:21:26.121109 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:21:26.121077    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-e7d45-predictor-bfcffd599-ntx59"]
Sep 17 18:21:26.121362 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:21:26.121326    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-e7d45-predictor-bfcffd599-ntx59" podUID="6fdf86e4-2b55-40a4-8ad2-385ee66aebc5" containerName="kserve-container" containerID="cri-o://a9120d85c96ac88a77ad7e8ba5edad1b311f474a9c9cbeeb730841c328c5b1b4" gracePeriod=30
Sep 17 18:21:26.121488 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:21:26.121405    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-e7d45-predictor-bfcffd599-ntx59" podUID="6fdf86e4-2b55-40a4-8ad2-385ee66aebc5" containerName="kube-rbac-proxy" containerID="cri-o://04eb1c1acf3f9adfef17101e3363cf521b9225536676680da1705c2435d03d2d" gracePeriod=30
Sep 17 18:21:26.173052 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:21:26.173019    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 18:21:26.384167 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:21:26.384135    2567 generic.go:356] "Generic (PLEG): container finished" podID="6fdf86e4-2b55-40a4-8ad2-385ee66aebc5" containerID="04eb1c1acf3f9adfef17101e3363cf521b9225536676680da1705c2435d03d2d" exitCode=2
Sep 17 18:21:26.384349 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:21:26.384213    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-e7d45-predictor-bfcffd599-ntx59" event={"ID":"6fdf86e4-2b55-40a4-8ad2-385ee66aebc5","Type":"ContainerDied","Data":"04eb1c1acf3f9adfef17101e3363cf521b9225536676680da1705c2435d03d2d"}
Sep 17 18:21:28.402359 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:21:28.402322    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-e7d45-7d4894f44c-9lvpm" podUID="2e9f62b6-51ab-469c-8b94-3839312c1653" containerName="splitter-graph-e7d45" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 18:21:29.146273 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:21:29.146228    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-e7d45-predictor-bfcffd599-ntx59" podUID="6fdf86e4-2b55-40a4-8ad2-385ee66aebc5" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.43:8643/healthz\": dial tcp 10.134.0.43:8643: connect: connection refused"
Sep 17 18:21:29.152286 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:21:29.152258    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-e7d45-predictor-bfcffd599-ntx59" podUID="6fdf86e4-2b55-40a4-8ad2-385ee66aebc5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Sep 17 18:21:32.411562 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:21:32.411501    2567 generic.go:356] "Generic (PLEG): container finished" podID="6fdf86e4-2b55-40a4-8ad2-385ee66aebc5" containerID="a9120d85c96ac88a77ad7e8ba5edad1b311f474a9c9cbeeb730841c328c5b1b4" exitCode=0
Sep 17 18:21:32.411927 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:21:32.411615    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-e7d45-predictor-bfcffd599-ntx59" event={"ID":"6fdf86e4-2b55-40a4-8ad2-385ee66aebc5","Type":"ContainerDied","Data":"a9120d85c96ac88a77ad7e8ba5edad1b311f474a9c9cbeeb730841c328c5b1b4"}
Sep 17 18:21:32.459474 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:21:32.459453    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-e7d45-predictor-bfcffd599-ntx59"
Sep 17 18:21:32.511014 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:21:32.510927    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6fdf86e4-2b55-40a4-8ad2-385ee66aebc5-proxy-tls\") pod \"6fdf86e4-2b55-40a4-8ad2-385ee66aebc5\" (UID: \"6fdf86e4-2b55-40a4-8ad2-385ee66aebc5\") "
Sep 17 18:21:32.511178 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:21:32.511047    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6fdf86e4-2b55-40a4-8ad2-385ee66aebc5-kube-rbac-proxy-sar-config\") pod \"6fdf86e4-2b55-40a4-8ad2-385ee66aebc5\" (UID: \"6fdf86e4-2b55-40a4-8ad2-385ee66aebc5\") "
Sep 17 18:21:32.511178 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:21:32.511111    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qmf74\" (UniqueName: \"kubernetes.io/projected/6fdf86e4-2b55-40a4-8ad2-385ee66aebc5-kube-api-access-qmf74\") pod \"6fdf86e4-2b55-40a4-8ad2-385ee66aebc5\" (UID: \"6fdf86e4-2b55-40a4-8ad2-385ee66aebc5\") "
Sep 17 18:21:32.511461 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:21:32.511432    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6fdf86e4-2b55-40a4-8ad2-385ee66aebc5-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "6fdf86e4-2b55-40a4-8ad2-385ee66aebc5" (UID: "6fdf86e4-2b55-40a4-8ad2-385ee66aebc5"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:21:32.512925 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:21:32.512901    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6fdf86e4-2b55-40a4-8ad2-385ee66aebc5-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "6fdf86e4-2b55-40a4-8ad2-385ee66aebc5" (UID: "6fdf86e4-2b55-40a4-8ad2-385ee66aebc5"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:21:32.513218 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:21:32.513198    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6fdf86e4-2b55-40a4-8ad2-385ee66aebc5-kube-api-access-qmf74" (OuterVolumeSpecName: "kube-api-access-qmf74") pod "6fdf86e4-2b55-40a4-8ad2-385ee66aebc5" (UID: "6fdf86e4-2b55-40a4-8ad2-385ee66aebc5"). InnerVolumeSpecName "kube-api-access-qmf74". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:21:32.612264 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:21:32.612222    2567 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6fdf86e4-2b55-40a4-8ad2-385ee66aebc5-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-237.ec2.internal\" DevicePath \"\""
Sep 17 18:21:32.612264 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:21:32.612258    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qmf74\" (UniqueName: \"kubernetes.io/projected/6fdf86e4-2b55-40a4-8ad2-385ee66aebc5-kube-api-access-qmf74\") on node \"ip-10-0-139-237.ec2.internal\" DevicePath \"\""
Sep 17 18:21:32.612264 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:21:32.612270    2567 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6fdf86e4-2b55-40a4-8ad2-385ee66aebc5-proxy-tls\") on node \"ip-10-0-139-237.ec2.internal\" DevicePath \"\""
Sep 17 18:21:33.402751 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:21:33.402711    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-e7d45-7d4894f44c-9lvpm" podUID="2e9f62b6-51ab-469c-8b94-3839312c1653" containerName="splitter-graph-e7d45" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 18:21:33.417788 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:21:33.417762    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-e7d45-predictor-bfcffd599-ntx59" event={"ID":"6fdf86e4-2b55-40a4-8ad2-385ee66aebc5","Type":"ContainerDied","Data":"48f36ddf5a429e24a6771bf5b14f8d5f45d298f7f25f9f7ab71f33a5e4009e82"}
Sep 17 18:21:33.418174 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:21:33.417821    2567 scope.go:117] "RemoveContainer" containerID="04eb1c1acf3f9adfef17101e3363cf521b9225536676680da1705c2435d03d2d"
Sep 17 18:21:33.418174 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:21:33.417769    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-e7d45-predictor-bfcffd599-ntx59"
Sep 17 18:21:33.425865 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:21:33.425847    2567 scope.go:117] "RemoveContainer" containerID="a9120d85c96ac88a77ad7e8ba5edad1b311f474a9c9cbeeb730841c328c5b1b4"
Sep 17 18:21:33.436719 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:21:33.436694    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-e7d45-predictor-bfcffd599-ntx59"]
Sep 17 18:21:33.438104 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:21:33.438086    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-e7d45-predictor-bfcffd599-ntx59"]
Sep 17 18:21:35.286322 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:21:35.286287    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6fdf86e4-2b55-40a4-8ad2-385ee66aebc5" path="/var/lib/kubelet/pods/6fdf86e4-2b55-40a4-8ad2-385ee66aebc5/volumes"
Sep 17 18:21:38.402714 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:21:38.402667    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-e7d45-7d4894f44c-9lvpm" podUID="2e9f62b6-51ab-469c-8b94-3839312c1653" containerName="splitter-graph-e7d45" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 18:21:38.403162 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:21:38.402793    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/splitter-graph-e7d45-7d4894f44c-9lvpm"
Sep 17 18:21:43.402609 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:21:43.402568    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-e7d45-7d4894f44c-9lvpm" podUID="2e9f62b6-51ab-469c-8b94-3839312c1653" containerName="splitter-graph-e7d45" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 18:21:48.402917 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:21:48.402875    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-e7d45-7d4894f44c-9lvpm" podUID="2e9f62b6-51ab-469c-8b94-3839312c1653" containerName="splitter-graph-e7d45" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 18:21:53.402224 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:21:53.402174    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-e7d45-7d4894f44c-9lvpm" podUID="2e9f62b6-51ab-469c-8b94-3839312c1653" containerName="splitter-graph-e7d45" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 18:21:56.108733 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:21:56.108709    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-e7d45-7d4894f44c-9lvpm"
Sep 17 18:21:56.210332 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:21:56.210304    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2e9f62b6-51ab-469c-8b94-3839312c1653-proxy-tls\") pod \"2e9f62b6-51ab-469c-8b94-3839312c1653\" (UID: \"2e9f62b6-51ab-469c-8b94-3839312c1653\") "
Sep 17 18:21:56.210501 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:21:56.210375    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2e9f62b6-51ab-469c-8b94-3839312c1653-openshift-service-ca-bundle\") pod \"2e9f62b6-51ab-469c-8b94-3839312c1653\" (UID: \"2e9f62b6-51ab-469c-8b94-3839312c1653\") "
Sep 17 18:21:56.210732 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:21:56.210707    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e9f62b6-51ab-469c-8b94-3839312c1653-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "2e9f62b6-51ab-469c-8b94-3839312c1653" (UID: "2e9f62b6-51ab-469c-8b94-3839312c1653"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:21:56.212306 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:21:56.212281    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e9f62b6-51ab-469c-8b94-3839312c1653-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "2e9f62b6-51ab-469c-8b94-3839312c1653" (UID: "2e9f62b6-51ab-469c-8b94-3839312c1653"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:21:56.310993 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:21:56.310968    2567 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2e9f62b6-51ab-469c-8b94-3839312c1653-proxy-tls\") on node \"ip-10-0-139-237.ec2.internal\" DevicePath \"\""
Sep 17 18:21:56.310993 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:21:56.310994    2567 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2e9f62b6-51ab-469c-8b94-3839312c1653-openshift-service-ca-bundle\") on node \"ip-10-0-139-237.ec2.internal\" DevicePath \"\""
Sep 17 18:21:56.512620 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:21:56.512579    2567 generic.go:356] "Generic (PLEG): container finished" podID="2e9f62b6-51ab-469c-8b94-3839312c1653" containerID="cbb6f39290aa63abcf54cfca98c7a6c609e91769d46d3fbe3d7a4e789f519304" exitCode=0
Sep 17 18:21:56.512811 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:21:56.512629    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-e7d45-7d4894f44c-9lvpm" event={"ID":"2e9f62b6-51ab-469c-8b94-3839312c1653","Type":"ContainerDied","Data":"cbb6f39290aa63abcf54cfca98c7a6c609e91769d46d3fbe3d7a4e789f519304"}
Sep 17 18:21:56.512811 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:21:56.512633    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-e7d45-7d4894f44c-9lvpm"
Sep 17 18:21:56.512811 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:21:56.512701    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-e7d45-7d4894f44c-9lvpm" event={"ID":"2e9f62b6-51ab-469c-8b94-3839312c1653","Type":"ContainerDied","Data":"823430a334ce67cec876f885b50278448f573955ef6b70659d9ac7127e205cfd"}
Sep 17 18:21:56.512811 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:21:56.512724    2567 scope.go:117] "RemoveContainer" containerID="cbb6f39290aa63abcf54cfca98c7a6c609e91769d46d3fbe3d7a4e789f519304"
Sep 17 18:21:56.521036 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:21:56.521019    2567 scope.go:117] "RemoveContainer" containerID="cbb6f39290aa63abcf54cfca98c7a6c609e91769d46d3fbe3d7a4e789f519304"
Sep 17 18:21:56.521450 ip-10-0-139-237 kubenswrapper[2567]: E0917 18:21:56.521431    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cbb6f39290aa63abcf54cfca98c7a6c609e91769d46d3fbe3d7a4e789f519304\": container with ID starting with cbb6f39290aa63abcf54cfca98c7a6c609e91769d46d3fbe3d7a4e789f519304 not found: ID does not exist" containerID="cbb6f39290aa63abcf54cfca98c7a6c609e91769d46d3fbe3d7a4e789f519304"
Sep 17 18:21:56.521519 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:21:56.521458    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cbb6f39290aa63abcf54cfca98c7a6c609e91769d46d3fbe3d7a4e789f519304"} err="failed to get container status \"cbb6f39290aa63abcf54cfca98c7a6c609e91769d46d3fbe3d7a4e789f519304\": rpc error: code = NotFound desc = could not find container \"cbb6f39290aa63abcf54cfca98c7a6c609e91769d46d3fbe3d7a4e789f519304\": container with ID starting with cbb6f39290aa63abcf54cfca98c7a6c609e91769d46d3fbe3d7a4e789f519304 not found: ID does not exist"
Sep 17 18:21:56.532708 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:21:56.532688    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-e7d45-7d4894f44c-9lvpm"]
Sep 17 18:21:56.536159 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:21:56.536138    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-e7d45-7d4894f44c-9lvpm"]
Sep 17 18:21:57.285686 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:21:57.285655    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2e9f62b6-51ab-469c-8b94-3839312c1653" path="/var/lib/kubelet/pods/2e9f62b6-51ab-469c-8b94-3839312c1653/volumes"
Sep 17 18:23:20.441929 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:23:20.441865    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 18:29:10.235443 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:29:10.235355    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-85bfa-7886b55cbc-fnw2h"]
Sep 17 18:29:10.238062 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:29:10.235568    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/switch-graph-85bfa-7886b55cbc-fnw2h" podUID="6d83c913-069b-43b9-af7a-b332934a9840" containerName="switch-graph-85bfa" containerID="cri-o://83475982114d8106c276bb05f8ec9548b3eec8f90b8cc821a58c84f602bb165f" gracePeriod=30
Sep 17 18:29:10.282969 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:29:10.282935    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 18:29:10.395745 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:29:10.395712    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-85bfa-predictor-5d4d479565-4dlks"]
Sep 17 18:29:10.395972 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:29:10.395944    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-85bfa-predictor-5d4d479565-4dlks" podUID="1219c9c7-e3bb-408a-99aa-7cc5583a5cdf" containerName="kserve-container" containerID="cri-o://0a648b015612347b5fd399cdea034a2c3f9324a8bc4a6c6580605ce8bec47f1c" gracePeriod=30
Sep 17 18:29:10.396063 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:29:10.396007    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-85bfa-predictor-5d4d479565-4dlks" podUID="1219c9c7-e3bb-408a-99aa-7cc5583a5cdf" containerName="kube-rbac-proxy" containerID="cri-o://81cf7aa2575afbccc42a0e353f7d7990cf7c4f6504c0aeb15f59af789144f783" gracePeriod=30
Sep 17 18:29:11.237809 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:29:11.237776    2567 generic.go:356] "Generic (PLEG): container finished" podID="1219c9c7-e3bb-408a-99aa-7cc5583a5cdf" containerID="81cf7aa2575afbccc42a0e353f7d7990cf7c4f6504c0aeb15f59af789144f783" exitCode=2
Sep 17 18:29:11.238177 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:29:11.237829    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-85bfa-predictor-5d4d479565-4dlks" event={"ID":"1219c9c7-e3bb-408a-99aa-7cc5583a5cdf","Type":"ContainerDied","Data":"81cf7aa2575afbccc42a0e353f7d7990cf7c4f6504c0aeb15f59af789144f783"}
Sep 17 18:29:11.397478 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:29:11.397442    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 18:29:11.956948 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:29:11.956914    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-b7rlv/must-gather-7f9ww"]
Sep 17 18:29:11.957247 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:29:11.957235    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6fdf86e4-2b55-40a4-8ad2-385ee66aebc5" containerName="kube-rbac-proxy"
Sep 17 18:29:11.957297 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:29:11.957250    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="6fdf86e4-2b55-40a4-8ad2-385ee66aebc5" containerName="kube-rbac-proxy"
Sep 17 18:29:11.957297 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:29:11.957271    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2e9f62b6-51ab-469c-8b94-3839312c1653" containerName="splitter-graph-e7d45"
Sep 17 18:29:11.957297 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:29:11.957277    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="2e9f62b6-51ab-469c-8b94-3839312c1653" containerName="splitter-graph-e7d45"
Sep 17 18:29:11.957297 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:29:11.957291    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6fdf86e4-2b55-40a4-8ad2-385ee66aebc5" containerName="kserve-container"
Sep 17 18:29:11.957297 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:29:11.957297    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="6fdf86e4-2b55-40a4-8ad2-385ee66aebc5" containerName="kserve-container"
Sep 17 18:29:11.957452 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:29:11.957345    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="2e9f62b6-51ab-469c-8b94-3839312c1653" containerName="splitter-graph-e7d45"
Sep 17 18:29:11.957452 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:29:11.957355    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="6fdf86e4-2b55-40a4-8ad2-385ee66aebc5" containerName="kserve-container"
Sep 17 18:29:11.957452 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:29:11.957361    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="6fdf86e4-2b55-40a4-8ad2-385ee66aebc5" containerName="kube-rbac-proxy"
Sep 17 18:29:11.960343 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:29:11.960328    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-b7rlv/must-gather-7f9ww"
Sep 17 18:29:11.962914 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:29:11.962890    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-b7rlv\"/\"openshift-service-ca.crt\""
Sep 17 18:29:11.963064 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:29:11.962951    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-b7rlv\"/\"default-dockercfg-nxv6g\""
Sep 17 18:29:11.963385 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:29:11.963363    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 18:29:11.964013 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:29:11.964000    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-b7rlv\"/\"kube-root-ca.crt\""
Sep 17 18:29:11.967920 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:29:11.967899    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-b7rlv/must-gather-7f9ww"]
Sep 17 18:29:12.061904 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:29:12.061841    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 18:29:12.096399 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:29:12.096371    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lc226\" (UniqueName: \"kubernetes.io/projected/c502413b-ef8d-4844-a473-1d664fcb7c3b-kube-api-access-lc226\") pod \"must-gather-7f9ww\" (UID: \"c502413b-ef8d-4844-a473-1d664fcb7c3b\") " pod="openshift-must-gather-b7rlv/must-gather-7f9ww"
Sep 17 18:29:12.096481 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:29:12.096414    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/c502413b-ef8d-4844-a473-1d664fcb7c3b-must-gather-output\") pod \"must-gather-7f9ww\" (UID: \"c502413b-ef8d-4844-a473-1d664fcb7c3b\") " pod="openshift-must-gather-b7rlv/must-gather-7f9ww"
Sep 17 18:29:12.197411 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:29:12.197378    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lc226\" (UniqueName: \"kubernetes.io/projected/c502413b-ef8d-4844-a473-1d664fcb7c3b-kube-api-access-lc226\") pod \"must-gather-7f9ww\" (UID: \"c502413b-ef8d-4844-a473-1d664fcb7c3b\") " pod="openshift-must-gather-b7rlv/must-gather-7f9ww"
Sep 17 18:29:12.197566 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:29:12.197424    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/c502413b-ef8d-4844-a473-1d664fcb7c3b-must-gather-output\") pod \"must-gather-7f9ww\" (UID: \"c502413b-ef8d-4844-a473-1d664fcb7c3b\") " pod="openshift-must-gather-b7rlv/must-gather-7f9ww"
Sep 17 18:29:12.197715 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:29:12.197700    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/c502413b-ef8d-4844-a473-1d664fcb7c3b-must-gather-output\") pod \"must-gather-7f9ww\" (UID: \"c502413b-ef8d-4844-a473-1d664fcb7c3b\") " pod="openshift-must-gather-b7rlv/must-gather-7f9ww"
Sep 17 18:29:12.205854 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:29:12.205825    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lc226\" (UniqueName: \"kubernetes.io/projected/c502413b-ef8d-4844-a473-1d664fcb7c3b-kube-api-access-lc226\") pod \"must-gather-7f9ww\" (UID: \"c502413b-ef8d-4844-a473-1d664fcb7c3b\") " pod="openshift-must-gather-b7rlv/must-gather-7f9ww"
Sep 17 18:29:12.280466 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:29:12.280395    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-b7rlv/must-gather-7f9ww"
Sep 17 18:29:12.550366 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:29:12.550336    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-b7rlv/must-gather-7f9ww"]
Sep 17 18:29:12.554364 ip-10-0-139-237 kubenswrapper[2567]: W0917 18:29:12.554337    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc502413b_ef8d_4844_a473_1d664fcb7c3b.slice/crio-483e2c232d3c7699e0e7e08d88c5bff6a706077ed217d408696c13387833b293 WatchSource:0}: Error finding container 483e2c232d3c7699e0e7e08d88c5bff6a706077ed217d408696c13387833b293: Status 404 returned error can't find the container with id 483e2c232d3c7699e0e7e08d88c5bff6a706077ed217d408696c13387833b293
Sep 17 18:29:12.556135 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:29:12.556119    2567 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 18:29:13.051706 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:29:13.051658    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-85bfa-predictor-5d4d479565-4dlks" podUID="1219c9c7-e3bb-408a-99aa-7cc5583a5cdf" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.41:8643/healthz\": dial tcp 10.134.0.41:8643: connect: connection refused"
Sep 17 18:29:13.250891 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:29:13.250853    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-b7rlv/must-gather-7f9ww" event={"ID":"c502413b-ef8d-4844-a473-1d664fcb7c3b","Type":"ContainerStarted","Data":"483e2c232d3c7699e0e7e08d88c5bff6a706077ed217d408696c13387833b293"}
Sep 17 18:29:14.330971 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:29:14.330926    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-85bfa-7886b55cbc-fnw2h" podUID="6d83c913-069b-43b9-af7a-b332934a9840" containerName="switch-graph-85bfa" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 18:29:16.702299 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:29:16.702257    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 18:29:18.280255 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:29:18.280220    2567 generic.go:356] "Generic (PLEG): container finished" podID="1219c9c7-e3bb-408a-99aa-7cc5583a5cdf" containerID="0a648b015612347b5fd399cdea034a2c3f9324a8bc4a6c6580605ce8bec47f1c" exitCode=0
Sep 17 18:29:18.280921 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:29:18.280308    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-85bfa-predictor-5d4d479565-4dlks" event={"ID":"1219c9c7-e3bb-408a-99aa-7cc5583a5cdf","Type":"ContainerDied","Data":"0a648b015612347b5fd399cdea034a2c3f9324a8bc4a6c6580605ce8bec47f1c"}
Sep 17 18:29:18.361000 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:29:18.360972    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-85bfa-predictor-5d4d479565-4dlks"
Sep 17 18:29:18.455726 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:29:18.455696    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1219c9c7-e3bb-408a-99aa-7cc5583a5cdf-proxy-tls\") pod \"1219c9c7-e3bb-408a-99aa-7cc5583a5cdf\" (UID: \"1219c9c7-e3bb-408a-99aa-7cc5583a5cdf\") "
Sep 17 18:29:18.455913 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:29:18.455749    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1219c9c7-e3bb-408a-99aa-7cc5583a5cdf-kube-rbac-proxy-sar-config\") pod \"1219c9c7-e3bb-408a-99aa-7cc5583a5cdf\" (UID: \"1219c9c7-e3bb-408a-99aa-7cc5583a5cdf\") "
Sep 17 18:29:18.455913 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:29:18.455828    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ts78w\" (UniqueName: \"kubernetes.io/projected/1219c9c7-e3bb-408a-99aa-7cc5583a5cdf-kube-api-access-ts78w\") pod \"1219c9c7-e3bb-408a-99aa-7cc5583a5cdf\" (UID: \"1219c9c7-e3bb-408a-99aa-7cc5583a5cdf\") "
Sep 17 18:29:18.456177 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:29:18.456143    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1219c9c7-e3bb-408a-99aa-7cc5583a5cdf-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "1219c9c7-e3bb-408a-99aa-7cc5583a5cdf" (UID: "1219c9c7-e3bb-408a-99aa-7cc5583a5cdf"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:29:18.458290 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:29:18.458266    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1219c9c7-e3bb-408a-99aa-7cc5583a5cdf-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "1219c9c7-e3bb-408a-99aa-7cc5583a5cdf" (UID: "1219c9c7-e3bb-408a-99aa-7cc5583a5cdf"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:29:18.458398 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:29:18.458307    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1219c9c7-e3bb-408a-99aa-7cc5583a5cdf-kube-api-access-ts78w" (OuterVolumeSpecName: "kube-api-access-ts78w") pod "1219c9c7-e3bb-408a-99aa-7cc5583a5cdf" (UID: "1219c9c7-e3bb-408a-99aa-7cc5583a5cdf"). InnerVolumeSpecName "kube-api-access-ts78w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:29:18.556557 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:29:18.556515    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ts78w\" (UniqueName: \"kubernetes.io/projected/1219c9c7-e3bb-408a-99aa-7cc5583a5cdf-kube-api-access-ts78w\") on node \"ip-10-0-139-237.ec2.internal\" DevicePath \"\""
Sep 17 18:29:18.556557 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:29:18.556557    2567 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1219c9c7-e3bb-408a-99aa-7cc5583a5cdf-proxy-tls\") on node \"ip-10-0-139-237.ec2.internal\" DevicePath \"\""
Sep 17 18:29:18.556745 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:29:18.556567    2567 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1219c9c7-e3bb-408a-99aa-7cc5583a5cdf-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-237.ec2.internal\" DevicePath \"\""
Sep 17 18:29:19.051723 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:29:19.051668    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-85bfa-predictor-5d4d479565-4dlks" podUID="1219c9c7-e3bb-408a-99aa-7cc5583a5cdf" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.41:8643/healthz\": context deadline exceeded"
Sep 17 18:29:19.057025 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:29:19.056989    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-85bfa-predictor-5d4d479565-4dlks" podUID="1219c9c7-e3bb-408a-99aa-7cc5583a5cdf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: i/o timeout"
Sep 17 18:29:19.293881 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:29:19.293829    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-b7rlv/must-gather-7f9ww" event={"ID":"c502413b-ef8d-4844-a473-1d664fcb7c3b","Type":"ContainerStarted","Data":"a5b16d86c1d213f2f14caa6fdaf66599cba477d5c0e86f641ab10d233ad6e173"}
Sep 17 18:29:19.294343 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:29:19.293888    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-b7rlv/must-gather-7f9ww" event={"ID":"c502413b-ef8d-4844-a473-1d664fcb7c3b","Type":"ContainerStarted","Data":"0d5fc8c899edb2f4948864a2cd2186dbfa3dbc56352070cb0c79568c331a8dcf"}
Sep 17 18:29:19.295662 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:29:19.295640    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-85bfa-predictor-5d4d479565-4dlks"
Sep 17 18:29:19.295662 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:29:19.295649    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-85bfa-predictor-5d4d479565-4dlks" event={"ID":"1219c9c7-e3bb-408a-99aa-7cc5583a5cdf","Type":"ContainerDied","Data":"1c76682e992c03949b42b45d69f5c6da117fd5b589ca8b84e558fd1b5d69002e"}
Sep 17 18:29:19.295818 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:29:19.295718    2567 scope.go:117] "RemoveContainer" containerID="81cf7aa2575afbccc42a0e353f7d7990cf7c4f6504c0aeb15f59af789144f783"
Sep 17 18:29:19.309421 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:29:19.309394    2567 scope.go:117] "RemoveContainer" containerID="0a648b015612347b5fd399cdea034a2c3f9324a8bc4a6c6580605ce8bec47f1c"
Sep 17 18:29:19.311451 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:29:19.311414    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-b7rlv/must-gather-7f9ww" podStartSLOduration=2.238645124 podStartE2EDuration="8.311400439s" totalImagesPullingTime="6.072755315s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:29:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:29:19.309670339 +0000 UTC m=+2900.333275477" watchObservedRunningTime="2026-09-17 18:29:19.311400439 +0000 UTC m=+2900.335005574"
Sep 17 18:29:19.322188 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:29:19.322166    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-85bfa-predictor-5d4d479565-4dlks"]
Sep 17 18:29:19.326136 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:29:19.324564    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-85bfa-predictor-5d4d479565-4dlks"]
Sep 17 18:29:19.330373 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:29:19.330345    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-85bfa-7886b55cbc-fnw2h" podUID="6d83c913-069b-43b9-af7a-b332934a9840" containerName="switch-graph-85bfa" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 18:29:21.287157 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:29:21.287127    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1219c9c7-e3bb-408a-99aa-7cc5583a5cdf" path="/var/lib/kubelet/pods/1219c9c7-e3bb-408a-99aa-7cc5583a5cdf/volumes"
Sep 17 18:29:24.328094 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:29:24.328059    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-85bfa-7886b55cbc-fnw2h" podUID="6d83c913-069b-43b9-af7a-b332934a9840" containerName="switch-graph-85bfa" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 18:29:24.328501 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:29:24.328185    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-85bfa-7886b55cbc-fnw2h"
Sep 17 18:29:27.539493 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:29:27.539450    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-85bfa-7886b55cbc-fnw2h_6d83c913-069b-43b9-af7a-b332934a9840/switch-graph-85bfa/0.log"
Sep 17 18:29:28.333127 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:29:28.333101    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-85bfa-7886b55cbc-fnw2h_6d83c913-069b-43b9-af7a-b332934a9840/switch-graph-85bfa/0.log"
Sep 17 18:29:29.131557 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:29:29.131483    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-85bfa-7886b55cbc-fnw2h_6d83c913-069b-43b9-af7a-b332934a9840/switch-graph-85bfa/0.log"
Sep 17 18:29:29.328768 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:29:29.328723    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-85bfa-7886b55cbc-fnw2h" podUID="6d83c913-069b-43b9-af7a-b332934a9840" containerName="switch-graph-85bfa" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 18:29:29.915643 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:29:29.915613    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-85bfa-7886b55cbc-fnw2h_6d83c913-069b-43b9-af7a-b332934a9840/switch-graph-85bfa/0.log"
Sep 17 18:29:30.688340 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:29:30.688308    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-85bfa-7886b55cbc-fnw2h_6d83c913-069b-43b9-af7a-b332934a9840/switch-graph-85bfa/0.log"
Sep 17 18:29:31.486614 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:29:31.486573    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-85bfa-7886b55cbc-fnw2h_6d83c913-069b-43b9-af7a-b332934a9840/switch-graph-85bfa/0.log"
Sep 17 18:29:32.290694 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:29:32.290664    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-85bfa-7886b55cbc-fnw2h_6d83c913-069b-43b9-af7a-b332934a9840/switch-graph-85bfa/0.log"
Sep 17 18:29:33.078144 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:29:33.078109    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-85bfa-7886b55cbc-fnw2h_6d83c913-069b-43b9-af7a-b332934a9840/switch-graph-85bfa/0.log"
Sep 17 18:29:33.870466 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:29:33.870425    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-85bfa-7886b55cbc-fnw2h_6d83c913-069b-43b9-af7a-b332934a9840/switch-graph-85bfa/0.log"
Sep 17 18:29:34.329458 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:29:34.329414    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-85bfa-7886b55cbc-fnw2h" podUID="6d83c913-069b-43b9-af7a-b332934a9840" containerName="switch-graph-85bfa" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 18:29:34.649347 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:29:34.649266    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-85bfa-7886b55cbc-fnw2h_6d83c913-069b-43b9-af7a-b332934a9840/switch-graph-85bfa/0.log"
Sep 17 18:29:35.419867 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:29:35.419839    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-85bfa-7886b55cbc-fnw2h_6d83c913-069b-43b9-af7a-b332934a9840/switch-graph-85bfa/0.log"
Sep 17 18:29:36.182943 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:29:36.182900    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-85bfa-7886b55cbc-fnw2h_6d83c913-069b-43b9-af7a-b332934a9840/switch-graph-85bfa/0.log"
Sep 17 18:29:36.980205 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:29:36.980174    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-85bfa-7886b55cbc-fnw2h_6d83c913-069b-43b9-af7a-b332934a9840/switch-graph-85bfa/0.log"
Sep 17 18:29:38.645071 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:29:38.645039    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-85bfa-7886b55cbc-fnw2h_6d83c913-069b-43b9-af7a-b332934a9840/switch-graph-85bfa/0.log"
Sep 17 18:29:39.329147 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:29:39.329109    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-85bfa-7886b55cbc-fnw2h" podUID="6d83c913-069b-43b9-af7a-b332934a9840" containerName="switch-graph-85bfa" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 18:29:39.501586 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:29:39.501556    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-85bfa-7886b55cbc-fnw2h_6d83c913-069b-43b9-af7a-b332934a9840/switch-graph-85bfa/0.log"
Sep 17 18:29:40.391354 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:29:40.391320    2567 generic.go:356] "Generic (PLEG): container finished" podID="6d83c913-069b-43b9-af7a-b332934a9840" containerID="83475982114d8106c276bb05f8ec9548b3eec8f90b8cc821a58c84f602bb165f" exitCode=0
Sep 17 18:29:40.391825 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:29:40.391401    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-85bfa-7886b55cbc-fnw2h" event={"ID":"6d83c913-069b-43b9-af7a-b332934a9840","Type":"ContainerDied","Data":"83475982114d8106c276bb05f8ec9548b3eec8f90b8cc821a58c84f602bb165f"}
Sep 17 18:29:40.393167 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:29:40.393145    2567 generic.go:356] "Generic (PLEG): container finished" podID="c502413b-ef8d-4844-a473-1d664fcb7c3b" containerID="0d5fc8c899edb2f4948864a2cd2186dbfa3dbc56352070cb0c79568c331a8dcf" exitCode=0
Sep 17 18:29:40.393275 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:29:40.393213    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-b7rlv/must-gather-7f9ww" event={"ID":"c502413b-ef8d-4844-a473-1d664fcb7c3b","Type":"ContainerDied","Data":"0d5fc8c899edb2f4948864a2cd2186dbfa3dbc56352070cb0c79568c331a8dcf"}
Sep 17 18:29:40.393509 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:29:40.393495    2567 scope.go:117] "RemoveContainer" containerID="0d5fc8c899edb2f4948864a2cd2186dbfa3dbc56352070cb0c79568c331a8dcf"
Sep 17 18:29:40.883353 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:29:40.883329    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-85bfa-7886b55cbc-fnw2h"
Sep 17 18:29:40.955512 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:29:40.955487    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6d83c913-069b-43b9-af7a-b332934a9840-openshift-service-ca-bundle\") pod \"6d83c913-069b-43b9-af7a-b332934a9840\" (UID: \"6d83c913-069b-43b9-af7a-b332934a9840\") "
Sep 17 18:29:40.955669 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:29:40.955519    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6d83c913-069b-43b9-af7a-b332934a9840-proxy-tls\") pod \"6d83c913-069b-43b9-af7a-b332934a9840\" (UID: \"6d83c913-069b-43b9-af7a-b332934a9840\") "
Sep 17 18:29:40.955826 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:29:40.955805    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d83c913-069b-43b9-af7a-b332934a9840-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "6d83c913-069b-43b9-af7a-b332934a9840" (UID: "6d83c913-069b-43b9-af7a-b332934a9840"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:29:40.957520 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:29:40.957498    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6d83c913-069b-43b9-af7a-b332934a9840-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "6d83c913-069b-43b9-af7a-b332934a9840" (UID: "6d83c913-069b-43b9-af7a-b332934a9840"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:29:41.056831 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:29:41.056798    2567 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6d83c913-069b-43b9-af7a-b332934a9840-openshift-service-ca-bundle\") on node \"ip-10-0-139-237.ec2.internal\" DevicePath \"\""
Sep 17 18:29:41.056831 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:29:41.056828    2567 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6d83c913-069b-43b9-af7a-b332934a9840-proxy-tls\") on node \"ip-10-0-139-237.ec2.internal\" DevicePath \"\""
Sep 17 18:29:41.230858 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:29:41.230829    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-b7rlv_must-gather-7f9ww_c502413b-ef8d-4844-a473-1d664fcb7c3b/gather/0.log"
Sep 17 18:29:41.397945 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:29:41.397867    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-85bfa-7886b55cbc-fnw2h"
Sep 17 18:29:41.397945 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:29:41.397884    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-85bfa-7886b55cbc-fnw2h" event={"ID":"6d83c913-069b-43b9-af7a-b332934a9840","Type":"ContainerDied","Data":"86d85516b0dee6146e30260b2c75f1e5c5469937b69b6089bdc64762cc3cb3f5"}
Sep 17 18:29:41.397945 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:29:41.397938    2567 scope.go:117] "RemoveContainer" containerID="83475982114d8106c276bb05f8ec9548b3eec8f90b8cc821a58c84f602bb165f"
Sep 17 18:29:41.414829 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:29:41.414804    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-85bfa-7886b55cbc-fnw2h"]
Sep 17 18:29:41.418498 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:29:41.418474    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/switch-graph-85bfa-7886b55cbc-fnw2h"]
Sep 17 18:29:41.670874 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:29:41.670789    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 18:29:41.814018 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:29:41.813985    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-f44dh/must-gather-vfppn"]
Sep 17 18:29:41.814324 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:29:41.814311    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1219c9c7-e3bb-408a-99aa-7cc5583a5cdf" containerName="kserve-container"
Sep 17 18:29:41.814374 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:29:41.814326    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="1219c9c7-e3bb-408a-99aa-7cc5583a5cdf" containerName="kserve-container"
Sep 17 18:29:41.814374 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:29:41.814334    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6d83c913-069b-43b9-af7a-b332934a9840" containerName="switch-graph-85bfa"
Sep 17 18:29:41.814374 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:29:41.814342    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="6d83c913-069b-43b9-af7a-b332934a9840" containerName="switch-graph-85bfa"
Sep 17 18:29:41.814374 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:29:41.814365    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1219c9c7-e3bb-408a-99aa-7cc5583a5cdf" containerName="kube-rbac-proxy"
Sep 17 18:29:41.814374 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:29:41.814372    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="1219c9c7-e3bb-408a-99aa-7cc5583a5cdf" containerName="kube-rbac-proxy"
Sep 17 18:29:41.814524 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:29:41.814417    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="1219c9c7-e3bb-408a-99aa-7cc5583a5cdf" containerName="kube-rbac-proxy"
Sep 17 18:29:41.814524 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:29:41.814428    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="1219c9c7-e3bb-408a-99aa-7cc5583a5cdf" containerName="kserve-container"
Sep 17 18:29:41.814524 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:29:41.814441    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="6d83c913-069b-43b9-af7a-b332934a9840" containerName="switch-graph-85bfa"
Sep 17 18:29:41.819135 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:29:41.819093    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-f44dh/must-gather-vfppn"
Sep 17 18:29:41.822168 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:29:41.822111    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-f44dh\"/\"kube-root-ca.crt\""
Sep 17 18:29:41.822168 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:29:41.822141    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-f44dh\"/\"openshift-service-ca.crt\""
Sep 17 18:29:41.824257 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:29:41.824235    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-f44dh/must-gather-vfppn"]
Sep 17 18:29:41.831904 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:29:41.831882    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 18:29:41.863917 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:29:41.863890    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hhgk9\" (UniqueName: \"kubernetes.io/projected/876fb65c-fc4d-43e7-b2c4-ef5a4670d86d-kube-api-access-hhgk9\") pod \"must-gather-vfppn\" (UID: \"876fb65c-fc4d-43e7-b2c4-ef5a4670d86d\") " pod="openshift-must-gather-f44dh/must-gather-vfppn"
Sep 17 18:29:41.864072 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:29:41.863935    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/876fb65c-fc4d-43e7-b2c4-ef5a4670d86d-must-gather-output\") pod \"must-gather-vfppn\" (UID: \"876fb65c-fc4d-43e7-b2c4-ef5a4670d86d\") " pod="openshift-must-gather-f44dh/must-gather-vfppn"
Sep 17 18:29:41.933618 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:29:41.933507    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 18:29:41.964359 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:29:41.964329    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/876fb65c-fc4d-43e7-b2c4-ef5a4670d86d-must-gather-output\") pod \"must-gather-vfppn\" (UID: \"876fb65c-fc4d-43e7-b2c4-ef5a4670d86d\") " pod="openshift-must-gather-f44dh/must-gather-vfppn"
Sep 17 18:29:41.964497 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:29:41.964409    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hhgk9\" (UniqueName: \"kubernetes.io/projected/876fb65c-fc4d-43e7-b2c4-ef5a4670d86d-kube-api-access-hhgk9\") pod \"must-gather-vfppn\" (UID: \"876fb65c-fc4d-43e7-b2c4-ef5a4670d86d\") " pod="openshift-must-gather-f44dh/must-gather-vfppn"
Sep 17 18:29:41.964685 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:29:41.964667    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/876fb65c-fc4d-43e7-b2c4-ef5a4670d86d-must-gather-output\") pod \"must-gather-vfppn\" (UID: \"876fb65c-fc4d-43e7-b2c4-ef5a4670d86d\") " pod="openshift-must-gather-f44dh/must-gather-vfppn"
Sep 17 18:29:41.972098 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:29:41.972080    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hhgk9\" (UniqueName: \"kubernetes.io/projected/876fb65c-fc4d-43e7-b2c4-ef5a4670d86d-kube-api-access-hhgk9\") pod \"must-gather-vfppn\" (UID: \"876fb65c-fc4d-43e7-b2c4-ef5a4670d86d\") " pod="openshift-must-gather-f44dh/must-gather-vfppn"
Sep 17 18:29:42.130114 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:29:42.130080    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-f44dh/must-gather-vfppn"
Sep 17 18:29:42.400151 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:29:42.400126    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-f44dh/must-gather-vfppn"]
Sep 17 18:29:42.402028 ip-10-0-139-237 kubenswrapper[2567]: W0917 18:29:42.401986    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod876fb65c_fc4d_43e7_b2c4_ef5a4670d86d.slice/crio-5ab8afdfc7535fa904a446cee9d59be6ed76b74f449d4d1ca45b18ab01ee7c95 WatchSource:0}: Error finding container 5ab8afdfc7535fa904a446cee9d59be6ed76b74f449d4d1ca45b18ab01ee7c95: Status 404 returned error can't find the container with id 5ab8afdfc7535fa904a446cee9d59be6ed76b74f449d4d1ca45b18ab01ee7c95
Sep 17 18:29:43.286914 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:29:43.286884    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6d83c913-069b-43b9-af7a-b332934a9840" path="/var/lib/kubelet/pods/6d83c913-069b-43b9-af7a-b332934a9840/volumes"
Sep 17 18:29:43.412823 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:29:43.412788    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-f44dh/must-gather-vfppn" event={"ID":"876fb65c-fc4d-43e7-b2c4-ef5a4670d86d","Type":"ContainerStarted","Data":"f15267ae9d4e4d1eb1fe7be1752e44b05d1e4c972245044cda495d05c34d9c16"}
Sep 17 18:29:43.413303 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:29:43.412842    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-f44dh/must-gather-vfppn" event={"ID":"876fb65c-fc4d-43e7-b2c4-ef5a4670d86d","Type":"ContainerStarted","Data":"5ab8afdfc7535fa904a446cee9d59be6ed76b74f449d4d1ca45b18ab01ee7c95"}
Sep 17 18:29:44.419311 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:29:44.419273    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-f44dh/must-gather-vfppn" event={"ID":"876fb65c-fc4d-43e7-b2c4-ef5a4670d86d","Type":"ContainerStarted","Data":"9c2e9f1356d6dd4f6d616242191d0f64f123daf7c0928a681d671648ecc65f66"}
Sep 17 18:29:44.437281 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:29:44.437221    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-f44dh/must-gather-vfppn" podStartSLOduration=2.561558673 podStartE2EDuration="3.437199722s" totalImagesPullingTime="875.641049ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:29:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:29:44.43429105 +0000 UTC m=+2925.457896180" watchObservedRunningTime="2026-09-17 18:29:44.437199722 +0000 UTC m=+2925.460804859"
Sep 17 18:29:44.725172 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:29:44.725091    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-vjstm_db73a8fe-d34c-41c8-8ba4-8040b4bacf13/global-pull-secret-syncer/0.log"
Sep 17 18:29:44.763445 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:29:44.763416    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log"
Sep 17 18:29:44.899005 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:29:44.898973    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-139-237.ec2.internal_840f4093b21dc0fac911784fe7877e0e/haproxy/0.log"
Sep 17 18:29:46.658542 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:29:46.658489    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-must-gather-b7rlv/must-gather-7f9ww"]
Sep 17 18:29:46.659036 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:29:46.658758    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-must-gather-b7rlv/must-gather-7f9ww" podUID="c502413b-ef8d-4844-a473-1d664fcb7c3b" containerName="copy" containerID="cri-o://a5b16d86c1d213f2f14caa6fdaf66599cba477d5c0e86f641ab10d233ad6e173" gracePeriod=2
Sep 17 18:29:46.662983 ip-10-0-139-237 kubenswrapper[2567]: E0917 18:29:46.662877    2567 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-7f9ww\" is forbidden: User \"system:node:ip-10-0-139-237.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-b7rlv\": no relationship found between node 'ip-10-0-139-237.ec2.internal' and this object" podUID="c502413b-ef8d-4844-a473-1d664fcb7c3b" pod="openshift-must-gather-b7rlv/must-gather-7f9ww"
Sep 17 18:29:46.665002 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:29:46.664975    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-must-gather-b7rlv/must-gather-7f9ww"]
Sep 17 18:29:47.016440 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:29:47.016347    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-b7rlv_must-gather-7f9ww_c502413b-ef8d-4844-a473-1d664fcb7c3b/copy/0.log"
Sep 17 18:29:47.018492 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:29:47.018233    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-b7rlv/must-gather-7f9ww"
Sep 17 18:29:47.022772 ip-10-0-139-237 kubenswrapper[2567]: E0917 18:29:47.022714    2567 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-7f9ww\" is forbidden: User \"system:node:ip-10-0-139-237.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-b7rlv\": no relationship found between node 'ip-10-0-139-237.ec2.internal' and this object" podUID="c502413b-ef8d-4844-a473-1d664fcb7c3b" pod="openshift-must-gather-b7rlv/must-gather-7f9ww"
Sep 17 18:29:47.117660 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:29:47.115592    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/c502413b-ef8d-4844-a473-1d664fcb7c3b-must-gather-output\") pod \"c502413b-ef8d-4844-a473-1d664fcb7c3b\" (UID: \"c502413b-ef8d-4844-a473-1d664fcb7c3b\") "
Sep 17 18:29:47.117660 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:29:47.115705    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lc226\" (UniqueName: \"kubernetes.io/projected/c502413b-ef8d-4844-a473-1d664fcb7c3b-kube-api-access-lc226\") pod \"c502413b-ef8d-4844-a473-1d664fcb7c3b\" (UID: \"c502413b-ef8d-4844-a473-1d664fcb7c3b\") "
Sep 17 18:29:47.117660 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:29:47.117603    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c502413b-ef8d-4844-a473-1d664fcb7c3b-must-gather-output" (OuterVolumeSpecName: "must-gather-output") pod "c502413b-ef8d-4844-a473-1d664fcb7c3b" (UID: "c502413b-ef8d-4844-a473-1d664fcb7c3b"). InnerVolumeSpecName "must-gather-output". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:29:47.121227 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:29:47.121185    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c502413b-ef8d-4844-a473-1d664fcb7c3b-kube-api-access-lc226" (OuterVolumeSpecName: "kube-api-access-lc226") pod "c502413b-ef8d-4844-a473-1d664fcb7c3b" (UID: "c502413b-ef8d-4844-a473-1d664fcb7c3b"). InnerVolumeSpecName "kube-api-access-lc226". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:29:47.216946 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:29:47.216882    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lc226\" (UniqueName: \"kubernetes.io/projected/c502413b-ef8d-4844-a473-1d664fcb7c3b-kube-api-access-lc226\") on node \"ip-10-0-139-237.ec2.internal\" DevicePath \"\""
Sep 17 18:29:47.216946 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:29:47.216919    2567 reconciler_common.go:299] "Volume detached for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/c502413b-ef8d-4844-a473-1d664fcb7c3b-must-gather-output\") on node \"ip-10-0-139-237.ec2.internal\" DevicePath \"\""
Sep 17 18:29:47.315836 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:29:47.315794    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c502413b-ef8d-4844-a473-1d664fcb7c3b" path="/var/lib/kubelet/pods/c502413b-ef8d-4844-a473-1d664fcb7c3b/volumes"
Sep 17 18:29:47.437432 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:29:47.437405    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-b7rlv_must-gather-7f9ww_c502413b-ef8d-4844-a473-1d664fcb7c3b/copy/0.log"
Sep 17 18:29:47.438100 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:29:47.438074    2567 generic.go:356] "Generic (PLEG): container finished" podID="c502413b-ef8d-4844-a473-1d664fcb7c3b" containerID="a5b16d86c1d213f2f14caa6fdaf66599cba477d5c0e86f641ab10d233ad6e173" exitCode=143
Sep 17 18:29:47.438335 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:29:47.438324    2567 scope.go:117] "RemoveContainer" containerID="a5b16d86c1d213f2f14caa6fdaf66599cba477d5c0e86f641ab10d233ad6e173"
Sep 17 18:29:47.438609 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:29:47.438593    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-b7rlv/must-gather-7f9ww"
Sep 17 18:29:47.452876 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:29:47.452852    2567 scope.go:117] "RemoveContainer" containerID="0d5fc8c899edb2f4948864a2cd2186dbfa3dbc56352070cb0c79568c331a8dcf"
Sep 17 18:29:47.477178 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:29:47.476898    2567 scope.go:117] "RemoveContainer" containerID="a5b16d86c1d213f2f14caa6fdaf66599cba477d5c0e86f641ab10d233ad6e173"
Sep 17 18:29:47.477718 ip-10-0-139-237 kubenswrapper[2567]: E0917 18:29:47.477419    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a5b16d86c1d213f2f14caa6fdaf66599cba477d5c0e86f641ab10d233ad6e173\": container with ID starting with a5b16d86c1d213f2f14caa6fdaf66599cba477d5c0e86f641ab10d233ad6e173 not found: ID does not exist" containerID="a5b16d86c1d213f2f14caa6fdaf66599cba477d5c0e86f641ab10d233ad6e173"
Sep 17 18:29:47.477718 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:29:47.477459    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a5b16d86c1d213f2f14caa6fdaf66599cba477d5c0e86f641ab10d233ad6e173"} err="failed to get container status \"a5b16d86c1d213f2f14caa6fdaf66599cba477d5c0e86f641ab10d233ad6e173\": rpc error: code = NotFound desc = could not find container \"a5b16d86c1d213f2f14caa6fdaf66599cba477d5c0e86f641ab10d233ad6e173\": container with ID starting with a5b16d86c1d213f2f14caa6fdaf66599cba477d5c0e86f641ab10d233ad6e173 not found: ID does not exist"
Sep 17 18:29:47.477718 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:29:47.477486    2567 scope.go:117] "RemoveContainer" containerID="0d5fc8c899edb2f4948864a2cd2186dbfa3dbc56352070cb0c79568c331a8dcf"
Sep 17 18:29:47.478144 ip-10-0-139-237 kubenswrapper[2567]: E0917 18:29:47.478066    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d5fc8c899edb2f4948864a2cd2186dbfa3dbc56352070cb0c79568c331a8dcf\": container with ID starting with 0d5fc8c899edb2f4948864a2cd2186dbfa3dbc56352070cb0c79568c331a8dcf not found: ID does not exist" containerID="0d5fc8c899edb2f4948864a2cd2186dbfa3dbc56352070cb0c79568c331a8dcf"
Sep 17 18:29:47.478144 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:29:47.478094    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d5fc8c899edb2f4948864a2cd2186dbfa3dbc56352070cb0c79568c331a8dcf"} err="failed to get container status \"0d5fc8c899edb2f4948864a2cd2186dbfa3dbc56352070cb0c79568c331a8dcf\": rpc error: code = NotFound desc = could not find container \"0d5fc8c899edb2f4948864a2cd2186dbfa3dbc56352070cb0c79568c331a8dcf\": container with ID starting with 0d5fc8c899edb2f4948864a2cd2186dbfa3dbc56352070cb0c79568c331a8dcf not found: ID does not exist"
Sep 17 18:29:47.689991 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:29:47.689921    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 18:29:47.751681 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:29:47.751454    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j5jw8_9a0f326b-5e07-4829-bc1f-084edb7f3b17/konnectivity-agent/0.log" limit=1024
Sep 17 18:29:49.284127 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:29:49.284050    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-84c885658c-wg927_db8f376a-5499-4162-b13f-f552de771aae/kube-state-metrics/0.log"
Sep 17 18:29:49.309869 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:29:49.309835    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-84c885658c-wg927_db8f376a-5499-4162-b13f-f552de771aae/kube-rbac-proxy-main/0.log"
Sep 17 18:29:49.337560 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:29:49.337515    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-84c885658c-wg927_db8f376a-5499-4162-b13f-f552de771aae/kube-rbac-proxy-self/0.log"
Sep 17 18:29:49.608440 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:29:49.608414    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-wdlzx_92415019-ec6d-46b5-99b0-0be3386db8bc/node-exporter/0.log"
Sep 17 18:29:49.635593 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:29:49.635559    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-wdlzx_92415019-ec6d-46b5-99b0-0be3386db8bc/kube-rbac-proxy/0.log"
Sep 17 18:29:49.657046 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:29:49.657018    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-wdlzx_92415019-ec6d-46b5-99b0-0be3386db8bc/init-textfile/0.log"
Sep 17 18:29:49.776185 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:29:49.776147    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_bfeb358d-808a-4246-a3d4-35add25eef8a/prometheus/0.log"
Sep 17 18:29:49.795172 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:29:49.795141    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_bfeb358d-808a-4246-a3d4-35add25eef8a/config-reloader/0.log"
Sep 17 18:29:49.818209 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:29:49.818174    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_bfeb358d-808a-4246-a3d4-35add25eef8a/thanos-sidecar/0.log"
Sep 17 18:29:49.841134 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:29:49.841108    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_bfeb358d-808a-4246-a3d4-35add25eef8a/kube-rbac-proxy-web/0.log"
Sep 17 18:29:49.864675 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:29:49.864608    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_bfeb358d-808a-4246-a3d4-35add25eef8a/kube-rbac-proxy/0.log"
Sep 17 18:29:49.887774 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:29:49.887751    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_bfeb358d-808a-4246-a3d4-35add25eef8a/kube-rbac-proxy-thanos/0.log"
Sep 17 18:29:49.909048 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:29:49.909016    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_bfeb358d-808a-4246-a3d4-35add25eef8a/init-config-reloader/0.log"
Sep 17 18:29:49.948471 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:29:49.948443    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-7bd5d56d6-v4vgx_14c1b35f-ef7b-463f-a407-4205465909ac/prometheus-operator/0.log"
Sep 17 18:29:49.964392 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:29:49.964350    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-7bd5d56d6-v4vgx_14c1b35f-ef7b-463f-a407-4205465909ac/kube-rbac-proxy/0.log"
Sep 17 18:29:49.990681 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:29:49.990652    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-5c6747dc7b-m66j7_026b6ea8-2b9e-4f09-97a1-dc77ef61a70c/prometheus-operator-admission-webhook/0.log"
Sep 17 18:29:50.025462 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:29:50.025428    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-8584f8444-pjmjp_d9aa6259-82fb-4485-8fc8-4afdf2a74d29/telemeter-client/0.log"
Sep 17 18:29:50.047075 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:29:50.047048    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-8584f8444-pjmjp_d9aa6259-82fb-4485-8fc8-4afdf2a74d29/reload/0.log"
Sep 17 18:29:50.069751 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:29:50.069721    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-8584f8444-pjmjp_d9aa6259-82fb-4485-8fc8-4afdf2a74d29/kube-rbac-proxy/0.log"
Sep 17 18:29:50.098890 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:29:50.098861    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-564fd7cf7d-hwkkx_38d1179d-e6c3-4949-bf01-b6ed37451005/thanos-query/0.log"
Sep 17 18:29:50.127823 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:29:50.127746    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-564fd7cf7d-hwkkx_38d1179d-e6c3-4949-bf01-b6ed37451005/kube-rbac-proxy-web/0.log"
Sep 17 18:29:50.148654 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:29:50.148623    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-564fd7cf7d-hwkkx_38d1179d-e6c3-4949-bf01-b6ed37451005/kube-rbac-proxy/0.log"
Sep 17 18:29:50.169711 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:29:50.169679    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-564fd7cf7d-hwkkx_38d1179d-e6c3-4949-bf01-b6ed37451005/prom-label-proxy/0.log"
Sep 17 18:29:50.190038 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:29:50.190013    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-564fd7cf7d-hwkkx_38d1179d-e6c3-4949-bf01-b6ed37451005/kube-rbac-proxy-rules/0.log"
Sep 17 18:29:50.211110 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:29:50.211083    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-564fd7cf7d-hwkkx_38d1179d-e6c3-4949-bf01-b6ed37451005/kube-rbac-proxy-metrics/0.log"
Sep 17 18:29:52.515203 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:29:52.515174    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-5f968588f4-6rgtl_cca22102-4b4b-46f7-8f37-fdced486b623/console/0.log"
Sep 17 18:29:53.210412 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:29:53.210379    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-f44dh/perf-node-gather-daemonset-q8bm6"]
Sep 17 18:29:53.210881 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:29:53.210857    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c502413b-ef8d-4844-a473-1d664fcb7c3b" containerName="gather"
Sep 17 18:29:53.211044 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:29:53.211032    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="c502413b-ef8d-4844-a473-1d664fcb7c3b" containerName="gather"
Sep 17 18:29:53.211141 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:29:53.211131    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c502413b-ef8d-4844-a473-1d664fcb7c3b" containerName="copy"
Sep 17 18:29:53.211226 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:29:53.211215    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="c502413b-ef8d-4844-a473-1d664fcb7c3b" containerName="copy"
Sep 17 18:29:53.211447 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:29:53.211418    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="c502413b-ef8d-4844-a473-1d664fcb7c3b" containerName="copy"
Sep 17 18:29:53.211447 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:29:53.211450    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="c502413b-ef8d-4844-a473-1d664fcb7c3b" containerName="gather"
Sep 17 18:29:53.214490 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:29:53.214467    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-f44dh/perf-node-gather-daemonset-q8bm6"
Sep 17 18:29:53.216977 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:29:53.216960    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-f44dh\"/\"default-dockercfg-mbd8r\""
Sep 17 18:29:53.221174 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:29:53.221152    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-f44dh/perf-node-gather-daemonset-q8bm6"]
Sep 17 18:29:53.277075 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:29:53.277026    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d0e203ff-2bd4-44d9-af08-6eeb09404fb2-proc\") pod \"perf-node-gather-daemonset-q8bm6\" (UID: \"d0e203ff-2bd4-44d9-af08-6eeb09404fb2\") " pod="openshift-must-gather-f44dh/perf-node-gather-daemonset-q8bm6"
Sep 17 18:29:53.277340 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:29:53.277322    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d0e203ff-2bd4-44d9-af08-6eeb09404fb2-sys\") pod \"perf-node-gather-daemonset-q8bm6\" (UID: \"d0e203ff-2bd4-44d9-af08-6eeb09404fb2\") " pod="openshift-must-gather-f44dh/perf-node-gather-daemonset-q8bm6"
Sep 17 18:29:53.277584 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:29:53.277514    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d0e203ff-2bd4-44d9-af08-6eeb09404fb2-podres\") pod \"perf-node-gather-daemonset-q8bm6\" (UID: \"d0e203ff-2bd4-44d9-af08-6eeb09404fb2\") " pod="openshift-must-gather-f44dh/perf-node-gather-daemonset-q8bm6"
Sep 17 18:29:53.277779 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:29:53.277753    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d0e203ff-2bd4-44d9-af08-6eeb09404fb2-lib-modules\") pod \"perf-node-gather-daemonset-q8bm6\" (UID: \"d0e203ff-2bd4-44d9-af08-6eeb09404fb2\") " pod="openshift-must-gather-f44dh/perf-node-gather-daemonset-q8bm6"
Sep 17 18:29:53.277954 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:29:53.277936    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-72b4r\" (UniqueName: \"kubernetes.io/projected/d0e203ff-2bd4-44d9-af08-6eeb09404fb2-kube-api-access-72b4r\") pod \"perf-node-gather-daemonset-q8bm6\" (UID: \"d0e203ff-2bd4-44d9-af08-6eeb09404fb2\") " pod="openshift-must-gather-f44dh/perf-node-gather-daemonset-q8bm6"
Sep 17 18:29:53.379612 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:29:53.379573    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d0e203ff-2bd4-44d9-af08-6eeb09404fb2-podres\") pod \"perf-node-gather-daemonset-q8bm6\" (UID: \"d0e203ff-2bd4-44d9-af08-6eeb09404fb2\") " pod="openshift-must-gather-f44dh/perf-node-gather-daemonset-q8bm6"
Sep 17 18:29:53.379872 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:29:53.379851    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d0e203ff-2bd4-44d9-af08-6eeb09404fb2-lib-modules\") pod \"perf-node-gather-daemonset-q8bm6\" (UID: \"d0e203ff-2bd4-44d9-af08-6eeb09404fb2\") " pod="openshift-must-gather-f44dh/perf-node-gather-daemonset-q8bm6"
Sep 17 18:29:53.379985 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:29:53.379944    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d0e203ff-2bd4-44d9-af08-6eeb09404fb2-podres\") pod \"perf-node-gather-daemonset-q8bm6\" (UID: \"d0e203ff-2bd4-44d9-af08-6eeb09404fb2\") " pod="openshift-must-gather-f44dh/perf-node-gather-daemonset-q8bm6"
Sep 17 18:29:53.380113 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:29:53.380021    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d0e203ff-2bd4-44d9-af08-6eeb09404fb2-lib-modules\") pod \"perf-node-gather-daemonset-q8bm6\" (UID: \"d0e203ff-2bd4-44d9-af08-6eeb09404fb2\") " pod="openshift-must-gather-f44dh/perf-node-gather-daemonset-q8bm6"
Sep 17 18:29:53.380113 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:29:53.380024    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-72b4r\" (UniqueName: \"kubernetes.io/projected/d0e203ff-2bd4-44d9-af08-6eeb09404fb2-kube-api-access-72b4r\") pod \"perf-node-gather-daemonset-q8bm6\" (UID: \"d0e203ff-2bd4-44d9-af08-6eeb09404fb2\") " pod="openshift-must-gather-f44dh/perf-node-gather-daemonset-q8bm6"
Sep 17 18:29:53.380263 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:29:53.380214    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d0e203ff-2bd4-44d9-af08-6eeb09404fb2-proc\") pod \"perf-node-gather-daemonset-q8bm6\" (UID: \"d0e203ff-2bd4-44d9-af08-6eeb09404fb2\") " pod="openshift-must-gather-f44dh/perf-node-gather-daemonset-q8bm6"
Sep 17 18:29:53.380263 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:29:53.380243    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d0e203ff-2bd4-44d9-af08-6eeb09404fb2-sys\") pod \"perf-node-gather-daemonset-q8bm6\" (UID: \"d0e203ff-2bd4-44d9-af08-6eeb09404fb2\") " pod="openshift-must-gather-f44dh/perf-node-gather-daemonset-q8bm6"
Sep 17 18:29:53.380336 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:29:53.380274    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d0e203ff-2bd4-44d9-af08-6eeb09404fb2-proc\") pod \"perf-node-gather-daemonset-q8bm6\" (UID: \"d0e203ff-2bd4-44d9-af08-6eeb09404fb2\") " pod="openshift-must-gather-f44dh/perf-node-gather-daemonset-q8bm6"
Sep 17 18:29:53.380336 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:29:53.380318    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d0e203ff-2bd4-44d9-af08-6eeb09404fb2-sys\") pod \"perf-node-gather-daemonset-q8bm6\" (UID: \"d0e203ff-2bd4-44d9-af08-6eeb09404fb2\") " pod="openshift-must-gather-f44dh/perf-node-gather-daemonset-q8bm6"
Sep 17 18:29:53.388230 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:29:53.388202    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-72b4r\" (UniqueName: \"kubernetes.io/projected/d0e203ff-2bd4-44d9-af08-6eeb09404fb2-kube-api-access-72b4r\") pod \"perf-node-gather-daemonset-q8bm6\" (UID: \"d0e203ff-2bd4-44d9-af08-6eeb09404fb2\") " pod="openshift-must-gather-f44dh/perf-node-gather-daemonset-q8bm6"
Sep 17 18:29:53.528657 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:29:53.528564    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-f44dh/perf-node-gather-daemonset-q8bm6"
Sep 17 18:29:53.820438 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:29:53.820343    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-f44dh/perf-node-gather-daemonset-q8bm6"]
Sep 17 18:29:53.823069 ip-10-0-139-237 kubenswrapper[2567]: W0917 18:29:53.823041    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podd0e203ff_2bd4_44d9_af08_6eeb09404fb2.slice/crio-dbfa4d36cff9ed8b8657375d2a040943d0f0d439317258bbee01e46522e07d9a WatchSource:0}: Error finding container dbfa4d36cff9ed8b8657375d2a040943d0f0d439317258bbee01e46522e07d9a: Status 404 returned error can't find the container with id dbfa4d36cff9ed8b8657375d2a040943d0f0d439317258bbee01e46522e07d9a
Sep 17 18:29:54.108570 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:29:54.108474    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-mdbmq_12911f3d-f3ef-4315-89d2-5760e9d6e938/dns/0.log"
Sep 17 18:29:54.128809 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:29:54.128779    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-mdbmq_12911f3d-f3ef-4315-89d2-5760e9d6e938/kube-rbac-proxy/0.log"
Sep 17 18:29:54.153332 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:29:54.153304    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-454w2_19204c4a-e05d-4159-94c8-477036fa461a/dns-node-resolver/0.log"
Sep 17 18:29:54.483502 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:29:54.483415    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-f44dh/perf-node-gather-daemonset-q8bm6" event={"ID":"d0e203ff-2bd4-44d9-af08-6eeb09404fb2","Type":"ContainerStarted","Data":"b04bb3b81ff901e411a8112baa7e58956ca7ac0a2b62a144f64afec57b758313"}
Sep 17 18:29:54.483502 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:29:54.483465    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-f44dh/perf-node-gather-daemonset-q8bm6" event={"ID":"d0e203ff-2bd4-44d9-af08-6eeb09404fb2","Type":"ContainerStarted","Data":"dbfa4d36cff9ed8b8657375d2a040943d0f0d439317258bbee01e46522e07d9a"}
Sep 17 18:29:54.483717 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:29:54.483508    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-f44dh/perf-node-gather-daemonset-q8bm6"
Sep 17 18:29:54.711008 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:29:54.710985    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-cfn7g_e45ad616-0131-4f4e-adf4-b567b76edc38/node-ca/0.log"
Sep 17 18:29:55.797434 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:29:55.797389    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-b9hsk_882622eb-219e-4d57-99b8-4c763d1b57b2/serve-healthcheck-canary/0.log"
Sep 17 18:29:56.267285 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:29:56.267213    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-4bpm9_ad8f026a-99c1-4153-a25f-e4638d09d341/kube-rbac-proxy/0.log"
Sep 17 18:29:56.286813 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:29:56.286783    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-4bpm9_ad8f026a-99c1-4153-a25f-e4638d09d341/exporter/0.log"
Sep 17 18:29:56.306565 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:29:56.306525    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-4bpm9_ad8f026a-99c1-4153-a25f-e4638d09d341/extractor/0.log"
Sep 17 18:29:58.715152 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:29:58.715125    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_model-serving-api-7b954ffc95-lt5tg_00a7cd51-8a31-4d58-87f2-16280e5e0774/server/0.log"
Sep 17 18:29:58.892643 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:29:58.892609    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-djfmf_eb97e72b-2549-4c22-889e-bacf215d6b26/s3-init/0.log"
Sep 17 18:29:58.907640 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:29:58.907609    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-djfmf_eb97e72b-2549-4c22-889e-bacf215d6b26/download-hf-model/0.log"
Sep 17 18:29:58.933180 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:29:58.933146    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-64568bcd49-8h4vd_ca127cca-a738-466b-a344-60e2f0943ae7/seaweedfs/0.log"
Sep 17 18:30:00.500062 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:30:00.500033    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-f44dh/perf-node-gather-daemonset-q8bm6"
Sep 17 18:30:00.516432 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:30:00.516388    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-f44dh/perf-node-gather-daemonset-q8bm6" podStartSLOduration=7.516371315 podStartE2EDuration="7.516371315s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 18:29:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:29:54.499655869 +0000 UTC m=+2935.523261030" watchObservedRunningTime="2026-09-17 18:30:00.516371315 +0000 UTC m=+2941.539976462"
Sep 17 18:30:04.546741 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:30:04.546711    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-ptjgb_bb03de28-d033-4586-8a80-54465decf01e/kube-multus-additional-cni-plugins/0.log"
Sep 17 18:30:04.566650 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:30:04.566626    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-ptjgb_bb03de28-d033-4586-8a80-54465decf01e/egress-router-binary-copy/0.log"
Sep 17 18:30:04.585583 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:30:04.585551    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-ptjgb_bb03de28-d033-4586-8a80-54465decf01e/cni-plugins/0.log"
Sep 17 18:30:04.605426 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:30:04.605400    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-ptjgb_bb03de28-d033-4586-8a80-54465decf01e/bond-cni-plugin/0.log"
Sep 17 18:30:04.624566 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:30:04.624524    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-ptjgb_bb03de28-d033-4586-8a80-54465decf01e/routeoverride-cni/0.log"
Sep 17 18:30:04.643131 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:30:04.643107    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-ptjgb_bb03de28-d033-4586-8a80-54465decf01e/whereabouts-cni-bincopy/0.log"
Sep 17 18:30:04.662485 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:30:04.662464    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-ptjgb_bb03de28-d033-4586-8a80-54465decf01e/whereabouts-cni/0.log"
Sep 17 18:30:04.875014 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:30:04.874987    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-dsxjr_41a677ee-5541-4579-a6e9-93b915b89d2d/kube-multus/0.log"
Sep 17 18:30:05.003797 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:30:05.003767    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-xg2wr_5ab7aae3-053b-4e36-9c4a-c523f0cfb615/network-metrics-daemon/0.log"
Sep 17 18:30:05.023222 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:30:05.023201    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-xg2wr_5ab7aae3-053b-4e36-9c4a-c523f0cfb615/kube-rbac-proxy/0.log"
Sep 17 18:30:06.120486 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:30:06.120460    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vwtvz_0a5f3756-68b8-4e1d-97fc-75a9e6e6fb27/ovn-controller/0.log"
Sep 17 18:30:06.135077 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:30:06.135053    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vwtvz_0a5f3756-68b8-4e1d-97fc-75a9e6e6fb27/ovn-acl-logging/0.log"
Sep 17 18:30:06.161101 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:30:06.161080    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vwtvz_0a5f3756-68b8-4e1d-97fc-75a9e6e6fb27/kube-rbac-proxy-node/0.log"
Sep 17 18:30:06.180216 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:30:06.180192    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vwtvz_0a5f3756-68b8-4e1d-97fc-75a9e6e6fb27/kube-rbac-proxy-ovn-metrics/0.log"
Sep 17 18:30:06.199649 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:30:06.199626    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vwtvz_0a5f3756-68b8-4e1d-97fc-75a9e6e6fb27/northd/0.log"
Sep 17 18:30:06.220214 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:30:06.220194    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vwtvz_0a5f3756-68b8-4e1d-97fc-75a9e6e6fb27/nbdb/0.log"
Sep 17 18:30:06.241487 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:30:06.241461    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vwtvz_0a5f3756-68b8-4e1d-97fc-75a9e6e6fb27/sbdb/0.log"
Sep 17 18:30:06.285101 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:30:06.285077    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vwtvz_0a5f3756-68b8-4e1d-97fc-75a9e6e6fb27/ovnkube-controller/0.log"
Sep 17 18:30:07.333745 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:30:07.333717    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-4cb4f_6d0a7973-cc86-4fda-9056-11b5923bd14c/network-check-target-container/0.log"
Sep 17 18:30:08.332212 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:30:08.332185    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-zrbgv_00f9603c-1639-4399-92ae-c1e4bf354395/iptables-alerter/0.log"
Sep 17 18:30:09.051641 ip-10-0-139-237 kubenswrapper[2567]: I0917 18:30:09.051608    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-node-tuning-operator_tuned-vvjl9_451c115d-30f1-4679-be41-ff6d2beedaf8/tuned/0.log"