Sep 17 23:10:15.149267 ip-10-0-142-4 systemd[1]: Starting Kubernetes Kubelet...
Sep 17 23:10:18.550345 ip-10-0-142-4 kubenswrapper[2584]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 17 23:10:18.550345 ip-10-0-142-4 kubenswrapper[2584]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 17 23:10:18.550345 ip-10-0-142-4 kubenswrapper[2584]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 17 23:10:18.550345 ip-10-0-142-4 kubenswrapper[2584]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 17 23:10:18.550345 ip-10-0-142-4 kubenswrapper[2584]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 17 23:10:18.551044 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:18.550429    2584 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 17 23:10:18.555464 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.555437    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 23:10:18.555464 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.555456    2584 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 23:10:18.555464 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.555460    2584 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 23:10:18.555464 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.555463    2584 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 23:10:18.555464 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.555466    2584 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 23:10:18.555464 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.555469    2584 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 23:10:18.555464 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.555472    2584 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 23:10:18.555710 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.555475    2584 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 23:10:18.555710 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.555477    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 23:10:18.555710 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.555480    2584 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 23:10:18.555710 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.555483    2584 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 23:10:18.555710 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.555485    2584 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 23:10:18.555710 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.555488    2584 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 23:10:18.555710 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.555491    2584 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 23:10:18.555710 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.555493    2584 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 23:10:18.555710 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.555496    2584 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 23:10:18.555710 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.555498    2584 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 23:10:18.555710 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.555501    2584 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 23:10:18.555710 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.555503    2584 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 23:10:18.555710 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.555506    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 23:10:18.555710 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.555508    2584 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 23:10:18.555710 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.555511    2584 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 23:10:18.555710 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.555514    2584 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 23:10:18.555710 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.555516    2584 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 23:10:18.555710 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.555519    2584 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 23:10:18.555710 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.555521    2584 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 23:10:18.555710 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.555523    2584 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 23:10:18.556197 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.555529    2584 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 23:10:18.556197 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.555531    2584 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 23:10:18.556197 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.555534    2584 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 23:10:18.556197 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.555536    2584 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 23:10:18.556197 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.555539    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 23:10:18.556197 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.555541    2584 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 23:10:18.556197 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.555543    2584 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 23:10:18.556197 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.555546    2584 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 23:10:18.556197 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.555548    2584 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 23:10:18.556197 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.555551    2584 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 23:10:18.556197 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.555553    2584 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 23:10:18.556197 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.555556    2584 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 23:10:18.556197 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.555558    2584 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 23:10:18.556197 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.555562    2584 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 23:10:18.556197 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.555564    2584 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 23:10:18.556197 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.555567    2584 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 23:10:18.556197 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.555569    2584 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 23:10:18.556197 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.555572    2584 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 23:10:18.556197 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.555574    2584 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 23:10:18.556197 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.555577    2584 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 23:10:18.556753 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.555581    2584 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 23:10:18.556753 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.555585    2584 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 23:10:18.556753 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.555588    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 23:10:18.556753 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.555590    2584 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 23:10:18.556753 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.555593    2584 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 23:10:18.556753 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.555596    2584 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 23:10:18.556753 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.555598    2584 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 23:10:18.556753 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.555601    2584 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 23:10:18.556753 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.555603    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 23:10:18.556753 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.555606    2584 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 23:10:18.556753 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.555609    2584 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 23:10:18.556753 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.555611    2584 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 23:10:18.556753 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.555613    2584 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 23:10:18.556753 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.555616    2584 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 23:10:18.556753 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.555618    2584 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 23:10:18.556753 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.555621    2584 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 23:10:18.556753 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.555625    2584 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 23:10:18.556753 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.555628    2584 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 23:10:18.556753 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.555631    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 23:10:18.557211 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.555634    2584 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 23:10:18.557211 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.555637    2584 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 23:10:18.557211 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.555640    2584 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 23:10:18.557211 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.555642    2584 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 23:10:18.557211 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.555645    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 23:10:18.557211 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.555647    2584 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 23:10:18.557211 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.555651    2584 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 23:10:18.557211 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.555654    2584 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 23:10:18.557211 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.555657    2584 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 23:10:18.557211 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.555659    2584 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 23:10:18.557211 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.555662    2584 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 23:10:18.557211 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.555664    2584 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 23:10:18.557211 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.555666    2584 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 23:10:18.557211 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.555669    2584 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 23:10:18.557211 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.555672    2584 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 23:10:18.557211 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.555675    2584 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 23:10:18.557211 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.555677    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 23:10:18.557211 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.555681    2584 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 23:10:18.557211 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.555683    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 23:10:18.557211 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.555686    2584 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 23:10:18.557693 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.555688    2584 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 23:10:18.557693 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.555691    2584 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 23:10:18.557693 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.555693    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 23:10:18.557693 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.555695    2584 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 23:10:18.557693 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.555698    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 23:10:18.557693 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.555700    2584 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 23:10:18.557693 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.555702    2584 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 23:10:18.557693 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.555705    2584 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 23:10:18.557693 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.555708    2584 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 23:10:18.557693 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.555712    2584 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 23:10:18.557693 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.555715    2584 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 23:10:18.557693 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.555718    2584 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 23:10:18.557693 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.555720    2584 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 23:10:18.557693 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.556414    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 23:10:18.557693 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.556421    2584 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 23:10:18.557693 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.556424    2584 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 23:10:18.557693 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.556427    2584 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 23:10:18.557693 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.556430    2584 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 23:10:18.557693 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.556434    2584 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 23:10:18.558198 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.556437    2584 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 23:10:18.558198 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.556439    2584 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 23:10:18.558198 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.556442    2584 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 23:10:18.558198 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.556445    2584 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 23:10:18.558198 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.556447    2584 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 23:10:18.558198 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.556450    2584 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 23:10:18.558198 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.556452    2584 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 23:10:18.558198 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.556455    2584 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 23:10:18.558198 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.556457    2584 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 23:10:18.558198 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.556459    2584 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 23:10:18.558198 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.556462    2584 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 23:10:18.558198 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.556465    2584 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 23:10:18.558198 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.556468    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 23:10:18.558198 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.556470    2584 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 23:10:18.558198 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.556472    2584 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 23:10:18.558198 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.556475    2584 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 23:10:18.558198 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.556478    2584 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 23:10:18.558198 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.556480    2584 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 23:10:18.558198 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.556482    2584 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 23:10:18.558198 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.556485    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 23:10:18.558696 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.556487    2584 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 23:10:18.558696 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.556489    2584 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 23:10:18.558696 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.556492    2584 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 23:10:18.558696 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.556495    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 23:10:18.558696 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.556497    2584 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 23:10:18.558696 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.556500    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 23:10:18.558696 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.556502    2584 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 23:10:18.558696 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.556505    2584 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 23:10:18.558696 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.556507    2584 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 23:10:18.558696 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.556509    2584 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 23:10:18.558696 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.556512    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 23:10:18.558696 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.556516    2584 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 23:10:18.558696 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.556518    2584 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 23:10:18.558696 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.556521    2584 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 23:10:18.558696 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.556524    2584 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 23:10:18.558696 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.556527    2584 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 23:10:18.558696 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.556529    2584 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 23:10:18.558696 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.556531    2584 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 23:10:18.558696 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.556536    2584 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 23:10:18.559178 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.556540    2584 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 23:10:18.559178 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.556543    2584 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 23:10:18.559178 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.556545    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 23:10:18.559178 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.556547    2584 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 23:10:18.559178 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.556550    2584 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 23:10:18.559178 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.556553    2584 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 23:10:18.559178 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.556556    2584 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 23:10:18.559178 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.556559    2584 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 23:10:18.559178 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.556563    2584 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 23:10:18.559178 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.556566    2584 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 23:10:18.559178 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.556569    2584 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 23:10:18.559178 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.556571    2584 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 23:10:18.559178 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.556574    2584 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 23:10:18.559178 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.556577    2584 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 23:10:18.559178 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.556580    2584 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 23:10:18.559178 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.556582    2584 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 23:10:18.559178 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.556585    2584 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 23:10:18.559178 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.556587    2584 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 23:10:18.559178 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.556590    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 23:10:18.559178 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.556592    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 23:10:18.559654 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.556595    2584 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 23:10:18.559654 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.556597    2584 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 23:10:18.559654 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.556600    2584 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 23:10:18.559654 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.556602    2584 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 23:10:18.559654 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.556606    2584 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 23:10:18.559654 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.556609    2584 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 23:10:18.559654 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.556611    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 23:10:18.559654 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.556614    2584 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 23:10:18.559654 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.556616    2584 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 23:10:18.559654 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.556620    2584 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 23:10:18.559654 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.556622    2584 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 23:10:18.559654 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.556624    2584 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 23:10:18.559654 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.556627    2584 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 23:10:18.559654 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.556629    2584 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 23:10:18.559654 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.556632    2584 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 23:10:18.559654 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.556634    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 23:10:18.559654 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.556636    2584 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 23:10:18.559654 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.556640    2584 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 23:10:18.559654 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.556644    2584 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 23:10:18.560137 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.556647    2584 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 23:10:18.560137 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.556650    2584 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 23:10:18.560137 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.556653    2584 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 23:10:18.560137 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.556656    2584 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 23:10:18.560137 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.556659    2584 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 23:10:18.560137 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.556662    2584 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 23:10:18.560137 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.556664    2584 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 23:10:18.560137 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.556667    2584 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 23:10:18.560137 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.556669    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 23:10:18.560137 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.556672    2584 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 23:10:18.560137 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.556674    2584 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 23:10:18.560137 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.556676    2584 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 23:10:18.560137 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.556679    2584 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 23:10:18.560137 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.556681    2584 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 23:10:18.560137 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.556683    2584 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 23:10:18.560137 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.557049    2584 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 23:10:18.560137 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.557054    2584 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 23:10:18.560137 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.557061    2584 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 23:10:18.560137 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.557065    2584 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 23:10:18.560137 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.557067    2584 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 23:10:18.560628 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.557070    2584 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 23:10:18.560628 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.557073    2584 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 23:10:18.560628 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.557075    2584 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 23:10:18.560628 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.557078    2584 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 23:10:18.560628 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.557082    2584 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 23:10:18.560628 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.557085    2584 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 23:10:18.560628 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.557088    2584 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 23:10:18.560628 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.557091    2584 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 23:10:18.560628 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.557093    2584 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 23:10:18.560628 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.557096    2584 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 23:10:18.560628 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.557099    2584 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 23:10:18.560628 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.557106    2584 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 23:10:18.560628 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.557108    2584 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 23:10:18.560628 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.557111    2584 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 23:10:18.560628 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.557113    2584 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 23:10:18.560628 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.557115    2584 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 23:10:18.560628 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.557118    2584 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 23:10:18.560628 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.557121    2584 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 23:10:18.560628 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.557123    2584 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 23:10:18.560628 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.557126    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 23:10:18.561132 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.557128    2584 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 23:10:18.561132 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.557131    2584 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 23:10:18.561132 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.557133    2584 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 23:10:18.561132 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.557136    2584 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 23:10:18.561132 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.557138    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 23:10:18.561132 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.557140    2584 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 23:10:18.561132 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.557143    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 23:10:18.561132 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.557145    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 23:10:18.561132 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.557147    2584 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 23:10:18.561132 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.557152    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 23:10:18.561132 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.557156    2584 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 23:10:18.561132 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.557159    2584 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 23:10:18.561132 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.557162    2584 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 23:10:18.561132 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.557165    2584 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 23:10:18.561132 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.557167    2584 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 23:10:18.561132 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.557170    2584 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 23:10:18.561132 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.557172    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 23:10:18.561132 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.557175    2584 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 23:10:18.561132 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.557177    2584 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 23:10:18.561617 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.557179    2584 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 23:10:18.561617 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.557182    2584 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 23:10:18.561617 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.557184    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 23:10:18.561617 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.557187    2584 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 23:10:18.561617 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.557189    2584 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 23:10:18.561617 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.557192    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 23:10:18.561617 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.557194    2584 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 23:10:18.561617 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.557197    2584 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 23:10:18.561617 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.557199    2584 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 23:10:18.561617 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.557202    2584 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 23:10:18.561617 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.557204    2584 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 23:10:18.561617 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.557206    2584 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 23:10:18.561617 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.557209    2584 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 23:10:18.561617 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.557211    2584 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 23:10:18.561617 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.557213    2584 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 23:10:18.561617 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.557217    2584 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 23:10:18.561617 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.557219    2584 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 23:10:18.561617 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.557222    2584 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 23:10:18.561617 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.557224    2584 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 23:10:18.561617 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.557227    2584 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 23:10:18.562128 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.557229    2584 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 23:10:18.562128 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.557232    2584 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 23:10:18.562128 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.557235    2584 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 23:10:18.562128 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.557238    2584 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 23:10:18.562128 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.557240    2584 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 23:10:18.562128 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.557243    2584 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 23:10:18.562128 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.557245    2584 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 23:10:18.562128 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.557249    2584 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 23:10:18.562128 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.557252    2584 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 23:10:18.562128 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.557254    2584 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 23:10:18.562128 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.557256    2584 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 23:10:18.562128 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.557261    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 23:10:18.562128 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.557263    2584 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 23:10:18.562128 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.557266    2584 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 23:10:18.562128 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.557268    2584 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 23:10:18.562128 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.557271    2584 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 23:10:18.562128 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.557273    2584 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 23:10:18.562128 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.557275    2584 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 23:10:18.562128 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.557278    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 23:10:18.562592 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.557280    2584 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 23:10:18.562592 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.557283    2584 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 23:10:18.562592 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.557285    2584 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 23:10:18.562592 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.557288    2584 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 23:10:18.562592 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.557290    2584 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 23:10:18.562592 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.557293    2584 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 23:10:18.562592 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.557295    2584 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 23:10:18.562592 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.557297    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 23:10:18.562592 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.557300    2584 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 23:10:18.562592 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.557302    2584 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 23:10:18.562592 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.557304    2584 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 23:10:18.562592 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.557307    2584 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 23:10:18.562592 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.557309    2584 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 23:10:18.562592 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.557311    2584 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 23:10:18.562592 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.557314    2584 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 23:10:18.562592 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.557317    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 23:10:18.562592 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:18.557387    2584 flags.go:64] FLAG: --address="0.0.0.0"
Sep 17 23:10:18.562592 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:18.557421    2584 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 17 23:10:18.562592 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:18.557427    2584 flags.go:64] FLAG: --anonymous-auth="true"
Sep 17 23:10:18.562592 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:18.557432    2584 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 17 23:10:18.562592 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:18.557436    2584 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 17 23:10:18.563156 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:18.557439    2584 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 17 23:10:18.563156 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:18.557443    2584 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 17 23:10:18.563156 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:18.557447    2584 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 17 23:10:18.563156 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:18.557450    2584 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 17 23:10:18.563156 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:18.557452    2584 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 17 23:10:18.563156 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:18.557456    2584 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 17 23:10:18.563156 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:18.557459    2584 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 17 23:10:18.563156 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:18.557461    2584 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 17 23:10:18.563156 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:18.557464    2584 flags.go:64] FLAG: --cgroup-root=""
Sep 17 23:10:18.563156 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:18.557467    2584 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 17 23:10:18.563156 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:18.557470    2584 flags.go:64] FLAG: --client-ca-file=""
Sep 17 23:10:18.563156 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:18.557473    2584 flags.go:64] FLAG: --cloud-provider="external"
Sep 17 23:10:18.563156 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:18.557476    2584 flags.go:64] FLAG: --cluster-dns="[]"
Sep 17 23:10:18.563156 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:18.557479    2584 flags.go:64] FLAG: --cluster-domain=""
Sep 17 23:10:18.563156 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:18.557482    2584 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 17 23:10:18.563156 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:18.557485    2584 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 17 23:10:18.563156 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:18.557488    2584 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 17 23:10:18.563156 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:18.557491    2584 flags.go:64] FLAG: --container-log-max-files="5"
Sep 17 23:10:18.563156 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:18.557494    2584 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 17 23:10:18.563156 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:18.557497    2584 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 17 23:10:18.563156 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:18.557500    2584 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 17 23:10:18.563156 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:18.557503    2584 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 17 23:10:18.563156 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:18.557506    2584 flags.go:64] FLAG: --contention-profiling="false"
Sep 17 23:10:18.563156 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:18.557508    2584 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 17 23:10:18.563709 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:18.557511    2584 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 17 23:10:18.563709 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:18.557514    2584 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 17 23:10:18.563709 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:18.557517    2584 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 17 23:10:18.563709 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:18.557521    2584 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 17 23:10:18.563709 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:18.557524    2584 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 17 23:10:18.563709 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:18.557527    2584 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 17 23:10:18.563709 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:18.557530    2584 flags.go:64] FLAG: --enable-load-reader="false"
Sep 17 23:10:18.563709 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:18.557532    2584 flags.go:64] FLAG: --enable-server="true"
Sep 17 23:10:18.563709 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:18.557535    2584 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 17 23:10:18.563709 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:18.557541    2584 flags.go:64] FLAG: --event-burst="100"
Sep 17 23:10:18.563709 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:18.557544    2584 flags.go:64] FLAG: --event-qps="50"
Sep 17 23:10:18.563709 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:18.557547    2584 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 17 23:10:18.563709 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:18.557550    2584 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 17 23:10:18.563709 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:18.557553    2584 flags.go:64] FLAG: --eviction-hard=""
Sep 17 23:10:18.563709 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:18.557556    2584 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 17 23:10:18.563709 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:18.557559    2584 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 17 23:10:18.563709 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:18.557561    2584 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 17 23:10:18.563709 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:18.557564    2584 flags.go:64] FLAG: --eviction-soft=""
Sep 17 23:10:18.563709 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:18.557567    2584 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 17 23:10:18.563709 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:18.557569    2584 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 17 23:10:18.563709 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:18.557572    2584 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 17 23:10:18.563709 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:18.557575    2584 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 17 23:10:18.563709 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:18.557578    2584 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 17 23:10:18.563709 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:18.557580    2584 flags.go:64] FLAG: --fail-swap-on="true"
Sep 17 23:10:18.563709 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:18.557583    2584 flags.go:64] FLAG: --feature-gates=""
Sep 17 23:10:18.564333 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:18.557587    2584 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 17 23:10:18.564333 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:18.557589    2584 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 17 23:10:18.564333 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:18.557592    2584 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 17 23:10:18.564333 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:18.557595    2584 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 17 23:10:18.564333 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:18.557598    2584 flags.go:64] FLAG: --healthz-port="10248"
Sep 17 23:10:18.564333 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:18.557601    2584 flags.go:64] FLAG: --help="false"
Sep 17 23:10:18.564333 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:18.557604    2584 flags.go:64] FLAG: --hostname-override="ip-10-0-142-4.ec2.internal"
Sep 17 23:10:18.564333 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:18.557606    2584 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 17 23:10:18.564333 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:18.557609    2584 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 17 23:10:18.564333 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:18.557612    2584 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 17 23:10:18.564333 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:18.557616    2584 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 17 23:10:18.564333 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:18.557619    2584 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 17 23:10:18.564333 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:18.557621    2584 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 17 23:10:18.564333 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:18.557624    2584 flags.go:64] FLAG: --image-service-endpoint=""
Sep 17 23:10:18.564333 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:18.557627    2584 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 17 23:10:18.564333 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:18.557629    2584 flags.go:64] FLAG: --kube-api-burst="100"
Sep 17 23:10:18.564333 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:18.557632    2584 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 17 23:10:18.564333 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:18.557635    2584 flags.go:64] FLAG: --kube-api-qps="50"
Sep 17 23:10:18.564333 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:18.557637    2584 flags.go:64] FLAG: --kube-reserved=""
Sep 17 23:10:18.564333 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:18.557640    2584 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 17 23:10:18.564333 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:18.557642    2584 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 17 23:10:18.564333 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:18.557645    2584 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 17 23:10:18.564333 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:18.557647    2584 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 17 23:10:18.564333 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:18.557650    2584 flags.go:64] FLAG: --lock-file=""
Sep 17 23:10:18.565184 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:18.557653    2584 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 17 23:10:18.565184 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:18.557655    2584 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 17 23:10:18.565184 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:18.557658    2584 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 17 23:10:18.565184 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:18.557663    2584 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 17 23:10:18.565184 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:18.557666    2584 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 17 23:10:18.565184 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:18.557669    2584 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 17 23:10:18.565184 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:18.557672    2584 flags.go:64] FLAG: --logging-format="text"
Sep 17 23:10:18.565184 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:18.557674    2584 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 17 23:10:18.565184 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:18.557678    2584 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 17 23:10:18.565184 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:18.557680    2584 flags.go:64] FLAG: --manifest-url=""
Sep 17 23:10:18.565184 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:18.557683    2584 flags.go:64] FLAG: --manifest-url-header=""
Sep 17 23:10:18.565184 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:18.557687    2584 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 17 23:10:18.565184 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:18.557690    2584 flags.go:64] FLAG: --max-open-files="1000000"
Sep 17 23:10:18.565184 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:18.557693    2584 flags.go:64] FLAG: --max-pods="110"
Sep 17 23:10:18.565184 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:18.557696    2584 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 17 23:10:18.565184 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:18.557699    2584 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 17 23:10:18.565184 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:18.557702    2584 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 17 23:10:18.565184 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:18.557706    2584 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 17 23:10:18.565184 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:18.557709    2584 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 17 23:10:18.565184 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:18.557713    2584 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 17 23:10:18.565184 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:18.557715    2584 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 17 23:10:18.565184 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:18.557722    2584 flags.go:64] FLAG: --node-status-max-images="50"
Sep 17 23:10:18.565184 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:18.557725    2584 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 17 23:10:18.565184 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:18.557728    2584 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 17 23:10:18.565184 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:18.557731    2584 flags.go:64] FLAG: --pod-cidr=""
Sep 17 23:10:18.566262 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:18.557733    2584 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a6abe77af61c5f2e3985a83e7d43538aefc38b1b0444127da14ee0e8cf8f76fc"
Sep 17 23:10:18.566262 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:18.557738    2584 flags.go:64] FLAG: --pod-manifest-path=""
Sep 17 23:10:18.566262 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:18.557741    2584 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 17 23:10:18.566262 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:18.557744    2584 flags.go:64] FLAG: --pods-per-core="0"
Sep 17 23:10:18.566262 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:18.557746    2584 flags.go:64] FLAG: --port="10250"
Sep 17 23:10:18.566262 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:18.557749    2584 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 17 23:10:18.566262 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:18.557752    2584 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-03dea6d04efe101c0"
Sep 17 23:10:18.566262 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:18.557755    2584 flags.go:64] FLAG: --qos-reserved=""
Sep 17 23:10:18.566262 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:18.557758    2584 flags.go:64] FLAG: --read-only-port="10255"
Sep 17 23:10:18.566262 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:18.557760    2584 flags.go:64] FLAG: --register-node="true"
Sep 17 23:10:18.566262 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:18.557763    2584 flags.go:64] FLAG: --register-with-taints=""
Sep 17 23:10:18.566262 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:18.557767    2584 flags.go:64] FLAG: --registry-burst="10"
Sep 17 23:10:18.566262 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:18.557770    2584 flags.go:64] FLAG: --registry-qps="5"
Sep 17 23:10:18.566262 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:18.557773    2584 flags.go:64] FLAG: --reserved-cpus=""
Sep 17 23:10:18.566262 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:18.557776    2584 flags.go:64] FLAG: --reserved-memory=""
Sep 17 23:10:18.566262 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:18.557779    2584 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 17 23:10:18.566262 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:18.557782    2584 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 17 23:10:18.566262 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:18.557785    2584 flags.go:64] FLAG: --rotate-certificates="false"
Sep 17 23:10:18.566262 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:18.557788    2584 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 17 23:10:18.566262 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:18.557791    2584 flags.go:64] FLAG: --runonce="false"
Sep 17 23:10:18.566262 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:18.557793    2584 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 17 23:10:18.566262 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:18.557796    2584 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 17 23:10:18.566262 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:18.557799    2584 flags.go:64] FLAG: --seccomp-default="false"
Sep 17 23:10:18.566262 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:18.557801    2584 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 17 23:10:18.566262 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:18.557806    2584 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 17 23:10:18.567371 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:18.557810    2584 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 17 23:10:18.567371 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:18.557813    2584 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 17 23:10:18.567371 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:18.557826    2584 flags.go:64] FLAG: --storage-driver-password="root"
Sep 17 23:10:18.567371 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:18.557829    2584 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 17 23:10:18.567371 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:18.557832    2584 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 17 23:10:18.567371 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:18.557835    2584 flags.go:64] FLAG: --storage-driver-user="root"
Sep 17 23:10:18.567371 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:18.557838    2584 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 17 23:10:18.567371 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:18.557841    2584 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 17 23:10:18.567371 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:18.557843    2584 flags.go:64] FLAG: --system-cgroups=""
Sep 17 23:10:18.567371 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:18.557846    2584 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 17 23:10:18.567371 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:18.557851    2584 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 17 23:10:18.567371 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:18.557854    2584 flags.go:64] FLAG: --tls-cert-file=""
Sep 17 23:10:18.567371 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:18.557856    2584 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 17 23:10:18.567371 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:18.557860    2584 flags.go:64] FLAG: --tls-min-version=""
Sep 17 23:10:18.567371 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:18.557862    2584 flags.go:64] FLAG: --tls-private-key-file=""
Sep 17 23:10:18.567371 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:18.557865    2584 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 17 23:10:18.567371 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:18.557867    2584 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 17 23:10:18.567371 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:18.557870    2584 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 17 23:10:18.567371 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:18.557873    2584 flags.go:64] FLAG: --v="2"
Sep 17 23:10:18.567371 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:18.557877    2584 flags.go:64] FLAG: --version="false"
Sep 17 23:10:18.567371 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:18.557884    2584 flags.go:64] FLAG: --vmodule=""
Sep 17 23:10:18.567371 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:18.557888    2584 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 17 23:10:18.567371 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:18.557891    2584 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 17 23:10:18.567371 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.558053    2584 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 23:10:18.567371 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.558058    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 23:10:18.568451 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.558061    2584 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 23:10:18.568451 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.558064    2584 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 23:10:18.568451 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.558067    2584 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 23:10:18.568451 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.558070    2584 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 23:10:18.568451 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.558073    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 23:10:18.568451 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.558076    2584 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 23:10:18.568451 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.558081    2584 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 23:10:18.568451 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.558084    2584 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 23:10:18.568451 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.558087    2584 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 23:10:18.568451 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.558090    2584 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 23:10:18.568451 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.558092    2584 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 23:10:18.568451 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.558095    2584 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 23:10:18.568451 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.558097    2584 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 23:10:18.568451 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.558100    2584 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 23:10:18.568451 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.558103    2584 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 23:10:18.568451 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.558107    2584 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 23:10:18.568451 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.558111    2584 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 23:10:18.568451 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.558114    2584 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 23:10:18.568451 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.558117    2584 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 23:10:18.568972 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.558119    2584 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 23:10:18.568972 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.558122    2584 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 23:10:18.568972 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.558125    2584 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 23:10:18.568972 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.558127    2584 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 23:10:18.568972 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.558130    2584 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 23:10:18.568972 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.558133    2584 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 23:10:18.568972 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.558135    2584 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 23:10:18.568972 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.558138    2584 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 23:10:18.568972 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.558140    2584 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 23:10:18.568972 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.558144    2584 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 23:10:18.568972 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.558146    2584 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 23:10:18.568972 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.558149    2584 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 23:10:18.568972 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.558151    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 23:10:18.568972 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.558154    2584 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 23:10:18.568972 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.558156    2584 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 23:10:18.568972 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.558159    2584 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 23:10:18.568972 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.558161    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 23:10:18.568972 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.558163    2584 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 23:10:18.568972 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.558166    2584 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 23:10:18.568972 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.558170    2584 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 23:10:18.569948 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.558174    2584 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 23:10:18.569948 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.558176    2584 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 23:10:18.569948 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.558179    2584 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 23:10:18.569948 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.558182    2584 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 23:10:18.569948 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.558184    2584 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 23:10:18.569948 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.558187    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 23:10:18.569948 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.558189    2584 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 23:10:18.569948 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.558192    2584 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 23:10:18.569948 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.558194    2584 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 23:10:18.569948 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.558197    2584 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 23:10:18.569948 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.558199    2584 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 23:10:18.569948 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.558202    2584 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 23:10:18.569948 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.558206    2584 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 23:10:18.569948 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.558210    2584 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 23:10:18.569948 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.558213    2584 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 23:10:18.569948 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.558216    2584 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 23:10:18.569948 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.558218    2584 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 23:10:18.569948 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.558221    2584 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 23:10:18.569948 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.558223    2584 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 23:10:18.570723 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.558226    2584 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 23:10:18.570723 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.558228    2584 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 23:10:18.570723 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.558231    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 23:10:18.570723 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.558234    2584 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 23:10:18.570723 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.558236    2584 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 23:10:18.570723 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.558239    2584 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 23:10:18.570723 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.558241    2584 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 23:10:18.570723 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.558244    2584 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 23:10:18.570723 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.558247    2584 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 23:10:18.570723 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.558249    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 23:10:18.570723 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.558252    2584 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 23:10:18.570723 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.558255    2584 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 23:10:18.570723 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.558259    2584 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 23:10:18.570723 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.558263    2584 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 23:10:18.570723 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.558265    2584 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 23:10:18.570723 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.558268    2584 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 23:10:18.570723 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.558270    2584 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 23:10:18.570723 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.558273    2584 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 23:10:18.570723 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.558275    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 23:10:18.570723 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.558278    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 23:10:18.571333 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.558280    2584 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 23:10:18.571333 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.558283    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 23:10:18.571333 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.558285    2584 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 23:10:18.571333 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.558288    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 23:10:18.571333 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.558290    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 23:10:18.571333 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.558293    2584 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 23:10:18.571333 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.558295    2584 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 23:10:18.571333 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.558299    2584 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 23:10:18.571333 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.558303    2584 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 23:10:18.571333 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.558305    2584 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 23:10:18.571333 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.558308    2584 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 23:10:18.571333 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.558311    2584 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 23:10:18.571333 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.558313    2584 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 23:10:18.571333 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.558316    2584 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 23:10:18.571333 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.558319    2584 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 23:10:18.571333 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.558322    2584 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 23:10:18.571333 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.558324    2584 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 23:10:18.571333 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.558327    2584 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 23:10:18.571333 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.558330    2584 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 23:10:18.571848 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:18.559034    2584 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 17 23:10:18.571848 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:18.567052    2584 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 17 23:10:18.571848 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:18.567070    2584 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 17 23:10:18.571848 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:18.567095    2584 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 17 23:10:18.571848 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:18.567108    2584 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 17 23:10:18.571848 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.567213    2584 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 23:10:18.571848 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.567220    2584 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 23:10:18.571848 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.567226    2584 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 23:10:18.571848 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.567230    2584 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 23:10:18.571848 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.567234    2584 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 23:10:18.571848 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.567238    2584 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 23:10:18.571848 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.567242    2584 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 23:10:18.571848 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.567246    2584 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 23:10:18.571848 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.567250    2584 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 23:10:18.571848 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.567254    2584 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 23:10:18.572291 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.567258    2584 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 23:10:18.572291 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.567263    2584 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 23:10:18.572291 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.567267    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 23:10:18.572291 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.567271    2584 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 23:10:18.572291 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.567275    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 23:10:18.572291 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.567280    2584 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 23:10:18.572291 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.567284    2584 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 23:10:18.572291 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.567288    2584 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 23:10:18.572291 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.567292    2584 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 23:10:18.572291 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.567298    2584 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 23:10:18.572291 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.567303    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 23:10:18.572291 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.567307    2584 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 23:10:18.572291 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.567310    2584 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 23:10:18.572291 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.567314    2584 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 23:10:18.572291 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.567326    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 23:10:18.572291 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.567330    2584 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 23:10:18.572291 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.567334    2584 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 23:10:18.572291 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.567338    2584 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 23:10:18.572291 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.567344    2584 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 23:10:18.572291 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.567350    2584 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 23:10:18.572790 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.567355    2584 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 23:10:18.572790 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.567359    2584 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 23:10:18.572790 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.567364    2584 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 23:10:18.572790 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.567368    2584 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 23:10:18.572790 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.567372    2584 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 23:10:18.572790 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.567375    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 23:10:18.572790 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.567379    2584 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 23:10:18.572790 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.567383    2584 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 23:10:18.572790 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.567387    2584 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 23:10:18.572790 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.567391    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 23:10:18.572790 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.567395    2584 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 23:10:18.572790 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.567398    2584 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 23:10:18.572790 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.567402    2584 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 23:10:18.572790 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.567407    2584 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 23:10:18.572790 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.567411    2584 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 23:10:18.572790 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.567415    2584 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 23:10:18.572790 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.567419    2584 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 23:10:18.572790 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.567423    2584 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 23:10:18.572790 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.567427    2584 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 23:10:18.573329 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.567430    2584 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 23:10:18.573329 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.567435    2584 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 23:10:18.573329 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.567438    2584 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 23:10:18.573329 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.567442    2584 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 23:10:18.573329 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.567446    2584 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 23:10:18.573329 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.567452    2584 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 23:10:18.573329 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.567458    2584 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 23:10:18.573329 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.567462    2584 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 23:10:18.573329 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.567475    2584 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 23:10:18.573329 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.567479    2584 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 23:10:18.573329 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.567483    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 23:10:18.573329 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.567487    2584 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 23:10:18.573329 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.567492    2584 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 23:10:18.573329 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.567496    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 23:10:18.573329 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.567501    2584 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 23:10:18.573329 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.567505    2584 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 23:10:18.573329 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.567509    2584 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 23:10:18.573329 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.567513    2584 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 23:10:18.573329 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.567517    2584 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 23:10:18.573775 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.567521    2584 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 23:10:18.573775 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.567525    2584 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 23:10:18.573775 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.567528    2584 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 23:10:18.573775 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.567532    2584 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 23:10:18.573775 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.567536    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 23:10:18.573775 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.567540    2584 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 23:10:18.573775 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.567544    2584 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 23:10:18.573775 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.567548    2584 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 23:10:18.573775 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.567551    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 23:10:18.573775 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.567555    2584 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 23:10:18.573775 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.567559    2584 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 23:10:18.573775 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.567563    2584 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 23:10:18.573775 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.567567    2584 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 23:10:18.573775 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.567571    2584 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 23:10:18.573775 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.567575    2584 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 23:10:18.573775 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.567578    2584 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 23:10:18.573775 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.567583    2584 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 23:10:18.573775 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.567586    2584 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 23:10:18.573775 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.567590    2584 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 23:10:18.573775 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.567594    2584 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 23:10:18.574447 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.567598    2584 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 23:10:18.574447 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.567602    2584 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 23:10:18.574447 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.567619    2584 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 23:10:18.574447 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.567624    2584 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 23:10:18.574447 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.567629    2584 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 23:10:18.574447 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.567633    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 23:10:18.574447 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.567637    2584 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 23:10:18.574447 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.567641    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 23:10:18.574447 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.567645    2584 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 23:10:18.574447 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.567649    2584 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 23:10:18.574447 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.567653    2584 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 23:10:18.574447 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:18.567660    2584 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 17 23:10:18.574447 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.567867    2584 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 23:10:18.574447 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.567874    2584 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 23:10:18.574447 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.567879    2584 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 23:10:18.574907 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.567888    2584 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 23:10:18.574907 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.567892    2584 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 23:10:18.574907 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.567896    2584 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 23:10:18.574907 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.567900    2584 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 23:10:18.574907 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.567905    2584 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 23:10:18.574907 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.567909    2584 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 23:10:18.574907 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.567915    2584 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 23:10:18.574907 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.567921    2584 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 23:10:18.574907 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.567925    2584 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 23:10:18.574907 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.567929    2584 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 23:10:18.574907 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.567933    2584 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 23:10:18.574907 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.567937    2584 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 23:10:18.574907 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.567941    2584 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 23:10:18.574907 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.567946    2584 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 23:10:18.574907 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.567950    2584 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 23:10:18.574907 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.567954    2584 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 23:10:18.574907 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.567957    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 23:10:18.574907 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.567962    2584 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 23:10:18.574907 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.567966    2584 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 23:10:18.574907 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.567969    2584 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 23:10:18.575742 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.567982    2584 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 23:10:18.575742 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.567987    2584 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 23:10:18.575742 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.567991    2584 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 23:10:18.575742 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.567996    2584 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 23:10:18.575742 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.567999    2584 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 23:10:18.575742 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.568021    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 23:10:18.575742 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.568025    2584 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 23:10:18.575742 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.568029    2584 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 23:10:18.575742 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.568032    2584 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 23:10:18.575742 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.568036    2584 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 23:10:18.575742 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.568040    2584 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 23:10:18.575742 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.568043    2584 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 23:10:18.575742 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.568047    2584 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 23:10:18.575742 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.568051    2584 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 23:10:18.575742 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.568055    2584 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 23:10:18.575742 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.568059    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 23:10:18.575742 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.568063    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 23:10:18.575742 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.568067    2584 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 23:10:18.575742 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.568071    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 23:10:18.575742 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.568074    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 23:10:18.576634 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.568078    2584 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 23:10:18.576634 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.568083    2584 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 23:10:18.576634 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.568087    2584 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 23:10:18.576634 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.568091    2584 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 23:10:18.576634 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.568095    2584 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 23:10:18.576634 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.568098    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 23:10:18.576634 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.568102    2584 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 23:10:18.576634 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.568106    2584 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 23:10:18.576634 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.568110    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 23:10:18.576634 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.568114    2584 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 23:10:18.576634 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.568118    2584 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 23:10:18.576634 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.568122    2584 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 23:10:18.576634 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.568126    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 23:10:18.576634 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.568139    2584 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 23:10:18.576634 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.568145    2584 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 23:10:18.576634 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.568151    2584 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 23:10:18.576634 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.568155    2584 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 23:10:18.576634 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.568160    2584 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 23:10:18.576634 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.568164    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 23:10:18.577257 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.568168    2584 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 23:10:18.577257 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.568171    2584 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 23:10:18.577257 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.568175    2584 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 23:10:18.577257 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.568179    2584 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 23:10:18.577257 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.568183    2584 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 23:10:18.577257 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.568187    2584 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 23:10:18.577257 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.568191    2584 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 23:10:18.577257 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.568195    2584 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 23:10:18.577257 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.568199    2584 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 23:10:18.577257 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.568203    2584 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 23:10:18.577257 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.568207    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 23:10:18.577257 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.568210    2584 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 23:10:18.577257 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.568215    2584 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 23:10:18.577257 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.568219    2584 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 23:10:18.577257 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.568223    2584 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 23:10:18.577257 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.568227    2584 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 23:10:18.577257 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.568231    2584 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 23:10:18.577257 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.568235    2584 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 23:10:18.577257 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.568238    2584 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 23:10:18.577932 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.568242    2584 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 23:10:18.577932 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.568246    2584 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 23:10:18.577932 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.568250    2584 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 23:10:18.577932 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.568253    2584 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 23:10:18.577932 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.568257    2584 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 23:10:18.577932 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.568269    2584 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 23:10:18.577932 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.568273    2584 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 23:10:18.577932 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.568277    2584 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 23:10:18.577932 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.568288    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 23:10:18.577932 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.568293    2584 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 23:10:18.577932 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.568297    2584 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 23:10:18.577932 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.568301    2584 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 23:10:18.577932 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.568304    2584 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 23:10:18.577932 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.568310    2584 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 23:10:18.577932 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.568315    2584 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 23:10:18.577932 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.568320    2584 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 23:10:18.577932 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.568324    2584 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 23:10:18.577932 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:18.568328    2584 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 23:10:18.578556 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:18.568336    2584 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 17 23:10:18.578556 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:18.568478    2584 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 17 23:10:18.578556 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:18.571236    2584 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 17 23:10:18.578556 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:18.572187    2584 server.go:1019] "Starting client certificate rotation"
Sep 17 23:10:18.578556 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:18.572289    2584 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 17 23:10:18.578556 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:18.572324    2584 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 17 23:10:18.578556 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:18.575470    2584 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 17 23:10:18.579912 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:18.579893    2584 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 17 23:10:18.587551 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:18.587535    2584 log.go:25] "Validated CRI v1 runtime API"
Sep 17 23:10:18.589036 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:18.588998    2584 log.go:25] "Validated CRI v1 image API"
Sep 17 23:10:18.589835 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:18.589810    2584 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 17 23:10:18.592655 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:18.592620    2584 fs.go:135] Filesystem UUIDs: map[75512f33-a30c-4e91-bd40-6a8282937fa4:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 d737c5e8-2ba1-435b-95d8-e87e307a6260:/dev/nvme0n1p4]
Sep 17 23:10:18.592718 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:18.592657    2584 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 17 23:10:18.596840 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:18.596740    2584 manager.go:216] Machine: {Timestamp:2026-09-17 23:10:18.596291541 +0000 UTC m=+1.961951448 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3201002 MemoryCapacity:32812154880 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2191357ee7681d8b9aa0e5bce3f9a0 SystemUUID:ec219135-7ee7-681d-8b9a-a0e5bce3f9a0 BootID:dc627eaa-a118-4837-aacb-f007e2dbab83 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406077440 Type:vfs Inodes:4005390 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562430976 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406077440 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:fa:e5:94:ff Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:fa:e5:94:ff Speed:0 Mtu:9001} {Name:ovs-system MacAddress:16:7d:3c:49:6b:0f Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812154880 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 17 23:10:18.596840 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:18.596838    2584 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 17 23:10:18.596944 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:18.596935    2584 manager.go:232] Version: {KernelVersion:5.14.0-570.141.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260914-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 17 23:10:18.598619 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:18.598596    2584 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 17 23:10:18.598752 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:18.598621    2584 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-142-4.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 17 23:10:18.599690 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:18.599680    2584 topology_manager.go:138] "Creating topology manager with none policy"
Sep 17 23:10:18.599722 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:18.599693    2584 container_manager_linux.go:309] "Creating device plugin manager"
Sep 17 23:10:18.599722 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:18.599705    2584 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 17 23:10:18.599722 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:18.599714    2584 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 17 23:10:18.599794 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:18.599784    2584 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 17 23:10:18.603636 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:18.603625    2584 state_mem.go:36] "Initialized new in-memory state store"
Sep 17 23:10:18.603755 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:18.603746    2584 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 17 23:10:18.603789 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:18.603780    2584 kubelet.go:486] "Attempting to sync node with API server"
Sep 17 23:10:18.603815 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:18.603794    2584 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 17 23:10:18.603815 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:18.603808    2584 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 17 23:10:18.603875 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:18.603817    2584 kubelet.go:398] "Adding apiserver pod source"
Sep 17 23:10:18.603875 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:18.603836    2584 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 17 23:10:18.604277 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:18.604261    2584 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 17 23:10:18.604318 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:18.604288    2584 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 17 23:10:18.606402 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:18.606383    2584 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.13-6.rhaos4.21.git6253162.el9" apiVersion="v1"
Sep 17 23:10:18.608413 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:18.608401    2584 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 17 23:10:18.608457 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:18.608429    2584 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 17 23:10:18.609920 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:18.609907    2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 17 23:10:18.609920 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:18.609923    2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 17 23:10:18.610044 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:18.609929    2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 17 23:10:18.610044 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:18.609934    2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 17 23:10:18.610044 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:18.609944    2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 17 23:10:18.610044 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:18.609949    2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 17 23:10:18.610044 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:18.609962    2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 17 23:10:18.610044 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:18.609967    2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 17 23:10:18.610044 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:18.609974    2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 17 23:10:18.610044 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:18.609979    2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 17 23:10:18.610044 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:18.609987    2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 17 23:10:18.610044 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:18.609992    2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 17 23:10:18.610044 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:18.610031    2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 17 23:10:18.610044 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:18.610037    2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 17 23:10:18.611821 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:18.611796    2584 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 17 23:10:18.613543 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:18.613532    2584 server.go:1268] "Started kubelet"
Sep 17 23:10:18.613686 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:18.613657    2584 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 17 23:10:18.613686 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:18.613628    2584 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 17 23:10:18.613765 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:18.613711    2584 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 17 23:10:18.646925 ip-10-0-142-4 systemd[1]: Started Kubernetes Kubelet.
Sep 17 23:10:18.647668 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:18.647644    2584 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 17 23:10:18.649144 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:18.649123    2584 server.go:310] "Adding debug handlers to kubelet server"
Sep 17 23:10:18.652327 ip-10-0-142-4 kubenswrapper[2584]: E0917 23:10:18.652296    2584 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 17 23:10:18.653304 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:18.653286    2584 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-142-4.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 17 23:10:18.653934 ip-10-0-142-4 kubenswrapper[2584]: E0917 23:10:18.653915    2584 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 17 23:10:18.653981 ip-10-0-142-4 kubenswrapper[2584]: E0917 23:10:18.653915    2584 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-142-4.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 17 23:10:18.654195 ip-10-0-142-4 kubenswrapper[2584]: E0917 23:10:18.653347    2584 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-142-4.ec2.internal.18d63e7f59ba6419  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-142-4.ec2.internal,UID:ip-10-0-142-4.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-142-4.ec2.internal,},FirstTimestamp:2026-09-17 23:10:18.613507097 +0000 UTC m=+1.979167004,LastTimestamp:2026-09-17 23:10:18.613507097 +0000 UTC m=+1.979167004,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-142-4.ec2.internal,}"
Sep 17 23:10:18.654800 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:18.654789    2584 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 17 23:10:18.654848 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:18.654806    2584 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 17 23:10:18.654896 ip-10-0-142-4 kubenswrapper[2584]: E0917 23:10:18.654862    2584 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-4.ec2.internal\" not found"
Sep 17 23:10:18.655407 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:18.655332    2584 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 17 23:10:18.655407 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:18.655334    2584 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 17 23:10:18.655541 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:18.655419    2584 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 17 23:10:18.655541 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:18.655511    2584 reconstruct.go:97] "Volume reconstruction finished"
Sep 17 23:10:18.655541 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:18.655520    2584 reconciler.go:29] "Reconciler: start to sync state"
Sep 17 23:10:18.655634 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:18.655614    2584 factory.go:55] Registering systemd factory
Sep 17 23:10:18.655634 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:18.655626    2584 factory.go:223] Registration of the systemd container factory successfully
Sep 17 23:10:18.655840 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:18.655828    2584 factory.go:153] Registering CRI-O factory
Sep 17 23:10:18.655925 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:18.655843    2584 factory.go:223] Registration of the crio container factory successfully
Sep 17 23:10:18.655925 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:18.655889    2584 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 17 23:10:18.655925 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:18.655906    2584 factory.go:103] Registering Raw factory
Sep 17 23:10:18.655925 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:18.655919    2584 manager.go:1196] Started watching for new ooms in manager
Sep 17 23:10:18.656635 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:18.656622    2584 manager.go:349] Starting recovery of all containers
Sep 17 23:10:18.657540 ip-10-0-142-4 kubenswrapper[2584]: E0917 23:10:18.657515    2584 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 17 23:10:18.657638 ip-10-0-142-4 kubenswrapper[2584]: E0917 23:10:18.657574    2584 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-142-4.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 17 23:10:18.666873 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:18.666858    2584 manager.go:354] Recovery completed
Sep 17 23:10:18.670912 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:18.670896    2584 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 23:10:18.672819 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:18.672800    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-4.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 23:10:18.672877 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:18.672832    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-4.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 23:10:18.672877 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:18.672849    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-4.ec2.internal" event="NodeHasSufficientPID"
Sep 17 23:10:18.673357 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:18.673343    2584 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 17 23:10:18.673357 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:18.673357    2584 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 17 23:10:18.673470 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:18.673373    2584 state_mem.go:36] "Initialized new in-memory state store"
Sep 17 23:10:18.673470 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:18.673398    2584 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 23:10:18.674119 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:18.674103    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-4.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 23:10:18.674160 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:18.674134    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-4.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 23:10:18.674160 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:18.674148    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-4.ec2.internal" event="NodeHasSufficientPID"
Sep 17 23:10:18.674953 ip-10-0-142-4 kubenswrapper[2584]: E0917 23:10:18.674869    2584 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-142-4.ec2.internal.18d63e7f5d4371b8  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-142-4.ec2.internal,UID:ip-10-0-142-4.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-142-4.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-142-4.ec2.internal,},FirstTimestamp:2026-09-17 23:10:18.672820664 +0000 UTC m=+2.038480581,LastTimestamp:2026-09-17 23:10:18.672820664 +0000 UTC m=+2.038480581,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-142-4.ec2.internal,}"
Sep 17 23:10:18.675486 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:18.675474    2584 policy_none.go:49] "None policy: Start"
Sep 17 23:10:18.675549 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:18.675491    2584 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 17 23:10:18.675549 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:18.675504    2584 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 17 23:10:18.677524 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:18.677511    2584 policy_none.go:47] "Start"
Sep 17 23:10:18.683119 ip-10-0-142-4 kubenswrapper[2584]: E0917 23:10:18.683055    2584 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-142-4.ec2.internal.18d63e7f5d43bc42  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-142-4.ec2.internal,UID:ip-10-0-142-4.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-142-4.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-142-4.ec2.internal,},FirstTimestamp:2026-09-17 23:10:18.672839746 +0000 UTC m=+2.038499654,LastTimestamp:2026-09-17 23:10:18.672839746 +0000 UTC m=+2.038499654,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-142-4.ec2.internal,}"
Sep 17 23:10:18.695819 ip-10-0-142-4 kubenswrapper[2584]: E0917 23:10:18.695747    2584 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-142-4.ec2.internal.18d63e7f5d43fb2d  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-142-4.ec2.internal,UID:ip-10-0-142-4.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-142-4.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-142-4.ec2.internal,},FirstTimestamp:2026-09-17 23:10:18.672855853 +0000 UTC m=+2.038515759,LastTimestamp:2026-09-17 23:10:18.672855853 +0000 UTC m=+2.038515759,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-142-4.ec2.internal,}"
Sep 17 23:10:18.701251 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:18.701143    2584 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-wkwrv"
Sep 17 23:10:18.708057 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:18.708042    2584 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-wkwrv"
Sep 17 23:10:18.709247 ip-10-0-142-4 kubenswrapper[2584]: E0917 23:10:18.709156    2584 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-142-4.ec2.internal.18d63e7f5d4371b8\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-142-4.ec2.internal.18d63e7f5d4371b8  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-142-4.ec2.internal,UID:ip-10-0-142-4.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-142-4.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-142-4.ec2.internal,},FirstTimestamp:2026-09-17 23:10:18.672820664 +0000 UTC m=+2.038480581,LastTimestamp:2026-09-17 23:10:18.674119778 +0000 UTC m=+2.039779685,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-142-4.ec2.internal,}"
Sep 17 23:10:18.728652 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:18.717569    2584 manager.go:337] "Starting Device Plugin manager"
Sep 17 23:10:18.728652 ip-10-0-142-4 kubenswrapper[2584]: E0917 23:10:18.717603    2584 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 17 23:10:18.728652 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:18.717616    2584 server.go:87] "Starting device plugin registration server"
Sep 17 23:10:18.728652 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:18.717849    2584 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 17 23:10:18.728652 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:18.717886    2584 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 17 23:10:18.728652 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:18.717967    2584 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 17 23:10:18.728652 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:18.718075    2584 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 17 23:10:18.728652 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:18.718086    2584 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 17 23:10:18.728652 ip-10-0-142-4 kubenswrapper[2584]: E0917 23:10:18.718562    2584 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 17 23:10:18.728652 ip-10-0-142-4 kubenswrapper[2584]: E0917 23:10:18.718610    2584 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-142-4.ec2.internal\" not found"
Sep 17 23:10:18.756135 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:18.756103    2584 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 17 23:10:18.757349 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:18.757336    2584 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 17 23:10:18.757407 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:18.757375    2584 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 17 23:10:18.757407 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:18.757393    2584 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 17 23:10:18.757481 ip-10-0-142-4 kubenswrapper[2584]: E0917 23:10:18.757424    2584 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 17 23:10:18.759638 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:18.759622    2584 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 23:10:18.818431 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:18.818384    2584 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 23:10:18.819225 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:18.819210    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-4.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 23:10:18.819307 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:18.819241    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-4.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 23:10:18.819307 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:18.819257    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-4.ec2.internal" event="NodeHasSufficientPID"
Sep 17 23:10:18.819307 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:18.819281    2584 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-142-4.ec2.internal"
Sep 17 23:10:18.828758 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:18.828745    2584 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-142-4.ec2.internal"
Sep 17 23:10:18.828813 ip-10-0-142-4 kubenswrapper[2584]: E0917 23:10:18.828762    2584 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-142-4.ec2.internal\": node \"ip-10-0-142-4.ec2.internal\" not found"
Sep 17 23:10:18.857686 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:18.857666    2584 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-4.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-142-4.ec2.internal"]
Sep 17 23:10:18.857773 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:18.857728    2584 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 23:10:18.859185 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:18.859166    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-4.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 23:10:18.859273 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:18.859193    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-4.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 23:10:18.859273 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:18.859202    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-4.ec2.internal" event="NodeHasSufficientPID"
Sep 17 23:10:18.860527 ip-10-0-142-4 kubenswrapper[2584]: E0917 23:10:18.860511    2584 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-4.ec2.internal\" not found"
Sep 17 23:10:18.861296 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:18.861285    2584 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 23:10:18.861469 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:18.861445    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-4.ec2.internal"
Sep 17 23:10:18.861528 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:18.861484    2584 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 23:10:18.861958 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:18.861944    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-4.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 23:10:18.861958 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:18.861952    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-4.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 23:10:18.862058 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:18.861972    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-4.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 23:10:18.862058 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:18.861983    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-4.ec2.internal" event="NodeHasSufficientPID"
Sep 17 23:10:18.862129 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:18.861973    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-4.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 23:10:18.862129 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:18.862094    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-4.ec2.internal" event="NodeHasSufficientPID"
Sep 17 23:10:18.864232 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:18.864219    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-4.ec2.internal"
Sep 17 23:10:18.864286 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:18.864241    2584 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 23:10:18.864865 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:18.864850    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-4.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 23:10:18.864926 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:18.864874    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-4.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 23:10:18.864926 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:18.864884    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-4.ec2.internal" event="NodeHasSufficientPID"
Sep 17 23:10:18.881508 ip-10-0-142-4 kubenswrapper[2584]: E0917 23:10:18.881490    2584 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-142-4.ec2.internal\" not found" node="ip-10-0-142-4.ec2.internal"
Sep 17 23:10:18.885327 ip-10-0-142-4 kubenswrapper[2584]: E0917 23:10:18.885311    2584 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-142-4.ec2.internal\" not found" node="ip-10-0-142-4.ec2.internal"
Sep 17 23:10:18.957466 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:18.957443    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/68edadcfa29cc8be7e22dcaea45c88a8-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-4.ec2.internal\" (UID: \"68edadcfa29cc8be7e22dcaea45c88a8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-4.ec2.internal"
Sep 17 23:10:18.957558 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:18.957517    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/68edadcfa29cc8be7e22dcaea45c88a8-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-4.ec2.internal\" (UID: \"68edadcfa29cc8be7e22dcaea45c88a8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-4.ec2.internal"
Sep 17 23:10:18.957558 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:18.957543    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/3771e4c220b5a223d0f484f25cdc29e8-config\") pod \"kube-apiserver-proxy-ip-10-0-142-4.ec2.internal\" (UID: \"3771e4c220b5a223d0f484f25cdc29e8\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-4.ec2.internal"
Sep 17 23:10:18.961552 ip-10-0-142-4 kubenswrapper[2584]: E0917 23:10:18.961539    2584 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-4.ec2.internal\" not found"
Sep 17 23:10:19.057641 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:19.057611    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/68edadcfa29cc8be7e22dcaea45c88a8-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-4.ec2.internal\" (UID: \"68edadcfa29cc8be7e22dcaea45c88a8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-4.ec2.internal"
Sep 17 23:10:19.057641 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:19.057643    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/3771e4c220b5a223d0f484f25cdc29e8-config\") pod \"kube-apiserver-proxy-ip-10-0-142-4.ec2.internal\" (UID: \"3771e4c220b5a223d0f484f25cdc29e8\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-4.ec2.internal"
Sep 17 23:10:19.057779 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:19.057659    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/68edadcfa29cc8be7e22dcaea45c88a8-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-4.ec2.internal\" (UID: \"68edadcfa29cc8be7e22dcaea45c88a8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-4.ec2.internal"
Sep 17 23:10:19.057779 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:19.057692    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/68edadcfa29cc8be7e22dcaea45c88a8-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-4.ec2.internal\" (UID: \"68edadcfa29cc8be7e22dcaea45c88a8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-4.ec2.internal"
Sep 17 23:10:19.057779 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:19.057723    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/3771e4c220b5a223d0f484f25cdc29e8-config\") pod \"kube-apiserver-proxy-ip-10-0-142-4.ec2.internal\" (UID: \"3771e4c220b5a223d0f484f25cdc29e8\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-4.ec2.internal"
Sep 17 23:10:19.057779 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:19.057721    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/68edadcfa29cc8be7e22dcaea45c88a8-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-4.ec2.internal\" (UID: \"68edadcfa29cc8be7e22dcaea45c88a8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-4.ec2.internal"
Sep 17 23:10:19.061727 ip-10-0-142-4 kubenswrapper[2584]: E0917 23:10:19.061714    2584 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-4.ec2.internal\" not found"
Sep 17 23:10:19.162549 ip-10-0-142-4 kubenswrapper[2584]: E0917 23:10:19.162506    2584 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-4.ec2.internal\" not found"
Sep 17 23:10:19.184660 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:19.184645    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-4.ec2.internal"
Sep 17 23:10:19.189445 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:19.189431    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-4.ec2.internal"
Sep 17 23:10:19.263571 ip-10-0-142-4 kubenswrapper[2584]: E0917 23:10:19.263542    2584 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-4.ec2.internal\" not found"
Sep 17 23:10:19.364024 ip-10-0-142-4 kubenswrapper[2584]: E0917 23:10:19.363987    2584 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-4.ec2.internal\" not found"
Sep 17 23:10:19.464541 ip-10-0-142-4 kubenswrapper[2584]: E0917 23:10:19.464496    2584 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-4.ec2.internal\" not found"
Sep 17 23:10:19.554271 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:19.554245    2584 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 23:10:19.565051 ip-10-0-142-4 kubenswrapper[2584]: E0917 23:10:19.565033    2584 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-4.ec2.internal\" not found"
Sep 17 23:10:19.572207 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:19.572181    2584 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 17 23:10:19.572324 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:19.572307    2584 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 17 23:10:19.572361 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:19.572340    2584 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 17 23:10:19.666114 ip-10-0-142-4 kubenswrapper[2584]: E0917 23:10:19.666076    2584 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-4.ec2.internal\" not found"
Sep 17 23:10:19.669593 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:19.669571    2584 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 23:10:19.715263 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:19.715191    2584 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-16 23:05:18 +0000 UTC" deadline="2028-05-24 20:09:32.525548723 +0000 UTC"
Sep 17 23:10:19.715263 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:19.715221    2584 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14756h59m12.810332762s"
Sep 17 23:10:19.755543 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:19.755521    2584 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-4.ec2.internal"
Sep 17 23:10:19.781632 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:19.781611    2584 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 17 23:10:19.782653 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:19.782640    2584 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-4.ec2.internal"
Sep 17 23:10:19.793396 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:19.793378    2584 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 17 23:10:19.828599 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:19.828560    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3771e4c220b5a223d0f484f25cdc29e8.slice/crio-4660d0d9e27e123f5aa0bc38fa39e98770d7949ae48447f34adbcc3aa2817032 WatchSource:0}: Error finding container 4660d0d9e27e123f5aa0bc38fa39e98770d7949ae48447f34adbcc3aa2817032: Status 404 returned error can't find the container with id 4660d0d9e27e123f5aa0bc38fa39e98770d7949ae48447f34adbcc3aa2817032
Sep 17 23:10:19.829069 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:19.829054    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68edadcfa29cc8be7e22dcaea45c88a8.slice/crio-3e6291a26948715d6e31e67de35259f57a966ca410921115cd8974aa88bd674c WatchSource:0}: Error finding container 3e6291a26948715d6e31e67de35259f57a966ca410921115cd8974aa88bd674c: Status 404 returned error can't find the container with id 3e6291a26948715d6e31e67de35259f57a966ca410921115cd8974aa88bd674c
Sep 17 23:10:19.834639 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:19.834627    2584 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 23:10:20.155985 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:20.155959    2584 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 23:10:20.605040 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:20.605015    2584 apiserver.go:52] "Watching apiserver"
Sep 17 23:10:20.612509 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:20.612481    2584 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 17 23:10:20.613030 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:20.612989    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-24tsm","kube-system/konnectivity-agent-g2vzq","openshift-cluster-node-tuning-operator/tuned-jf4g9","openshift-network-diagnostics/network-check-target-wgps9","openshift-network-operator/iptables-alerter-4bls7","openshift-ovn-kubernetes/ovnkube-node-tcglq","kube-system/kube-apiserver-proxy-ip-10-0-142-4.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jph2d","openshift-dns/node-resolver-895t6","openshift-image-registry/node-ca-5spz7","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-4.ec2.internal","openshift-multus/multus-9j2wk","openshift-multus/multus-additional-cni-plugins-2gjg8","openshift-multus/network-metrics-daemon-4bbdp"]
Sep 17 23:10:20.615574 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:20.615552    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-tcglq"
Sep 17 23:10:20.618086 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:20.618064    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-9j2wk"
Sep 17 23:10:20.618984 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:20.618955    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 17 23:10:20.619762 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:20.619739    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 17 23:10:20.619978 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:20.619954    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 17 23:10:20.620157 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:20.620143    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-gfdn5\""
Sep 17 23:10:20.620322 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:20.620306    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 17 23:10:20.620508 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:20.620493    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 17 23:10:20.620673 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:20.620659    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 17 23:10:20.621951 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:20.621605    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 17 23:10:20.621951 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:20.621639    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-2ljbs\""
Sep 17 23:10:20.621951 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:20.621852    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 17 23:10:20.621951 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:20.621905    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 17 23:10:20.622219 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:20.622100    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 17 23:10:20.622219 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:20.622123    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-jf4g9"
Sep 17 23:10:20.623724 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:20.623497    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wgps9"
Sep 17 23:10:20.623724 ip-10-0-142-4 kubenswrapper[2584]: E0917 23:10:20.623558    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wgps9" podUID="a2a26114-4c24-4dce-929c-3cb5caf17e6a"
Sep 17 23:10:20.624521 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:20.624502    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 17 23:10:20.624776 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:20.624760    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 17 23:10:20.625023 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:20.624954    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-4tdcq\""
Sep 17 23:10:20.625809 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:20.625789    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-4bls7"
Sep 17 23:10:20.628265 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:20.628220    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-24tsm"
Sep 17 23:10:20.628343 ip-10-0-142-4 kubenswrapper[2584]: E0917 23:10:20.628281    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-24tsm" podUID="fa08be4f-e950-4dee-a227-feb39c2ba7e0"
Sep 17 23:10:20.628676 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:20.628655    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 17 23:10:20.628796 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:20.628734    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 17 23:10:20.628796 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:20.628753    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-t2p6k\""
Sep 17 23:10:20.628796 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:20.628788    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 17 23:10:20.630857 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:20.630591    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jph2d"
Sep 17 23:10:20.633057 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:20.632999    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-w9xh8\""
Sep 17 23:10:20.633168 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:20.633137    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 17 23:10:20.633215 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:20.633175    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 17 23:10:20.633689 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:20.633672    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 17 23:10:20.635703 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:20.635345    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-895t6"
Sep 17 23:10:20.637721 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:20.637705    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-5spz7"
Sep 17 23:10:20.637807 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:20.637752    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 17 23:10:20.640348 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:20.640266    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-b8h9b\""
Sep 17 23:10:20.640476 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:20.640455    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 17 23:10:20.642142 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:20.640780    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-g2vzq"
Sep 17 23:10:20.642142 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:20.641808    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 17 23:10:20.643356 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:20.643318    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-2gjg8"
Sep 17 23:10:20.644163 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:20.644145    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 17 23:10:20.645192 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:20.645157    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 17 23:10:20.645490 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:20.645462    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-vjvs5\""
Sep 17 23:10:20.645695 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:20.645678    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4bbdp"
Sep 17 23:10:20.645777 ip-10-0-142-4 kubenswrapper[2584]: E0917 23:10:20.645739    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4bbdp" podUID="d672140f-ad59-4a68-9dde-80b62492af7f"
Sep 17 23:10:20.649033 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:20.648109    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 17 23:10:20.649033 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:20.648153    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 17 23:10:20.649033 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:20.648186    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-hzpmp\""
Sep 17 23:10:20.649033 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:20.648420    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 17 23:10:20.649033 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:20.648433    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-s8vjw\""
Sep 17 23:10:20.649033 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:20.648540    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 17 23:10:20.656095 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:20.656066    2584 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 17 23:10:20.656774 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:20.656758    2584 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 17 23:10:20.664564 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:20.664542    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/891302e3-91c1-4b56-a169-950bb8587aa5-run\") pod \"tuned-jf4g9\" (UID: \"891302e3-91c1-4b56-a169-950bb8587aa5\") " pod="openshift-cluster-node-tuning-operator/tuned-jf4g9"
Sep 17 23:10:20.664664 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:20.664578    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/1ae5e3b8-3cf3-4664-b502-8ab797838fb8-socket-dir\") pod \"aws-ebs-csi-driver-node-jph2d\" (UID: \"1ae5e3b8-3cf3-4664-b502-8ab797838fb8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jph2d"
Sep 17 23:10:20.664664 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:20.664602    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/1ae5e3b8-3cf3-4664-b502-8ab797838fb8-device-dir\") pod \"aws-ebs-csi-driver-node-jph2d\" (UID: \"1ae5e3b8-3cf3-4664-b502-8ab797838fb8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jph2d"
Sep 17 23:10:20.664664 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:20.664649    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ea8e5a0a-14f1-4be6-84e1-33567e5d6032-system-cni-dir\") pod \"multus-additional-cni-plugins-2gjg8\" (UID: \"ea8e5a0a-14f1-4be6-84e1-33567e5d6032\") " pod="openshift-multus/multus-additional-cni-plugins-2gjg8"
Sep 17 23:10:20.664809 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:20.664679    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/05765151-c799-4fe2-9f2e-cf4181e84ffa-konnectivity-ca\") pod \"konnectivity-agent-g2vzq\" (UID: \"05765151-c799-4fe2-9f2e-cf4181e84ffa\") " pod="kube-system/konnectivity-agent-g2vzq"
Sep 17 23:10:20.664809 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:20.664703    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/123417a3-f884-4c2f-b7ac-837813412743-run-ovn\") pod \"ovnkube-node-tcglq\" (UID: \"123417a3-f884-4c2f-b7ac-837813412743\") " pod="openshift-ovn-kubernetes/ovnkube-node-tcglq"
Sep 17 23:10:20.664809 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:20.664724    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/891302e3-91c1-4b56-a169-950bb8587aa5-etc-sysctl-d\") pod \"tuned-jf4g9\" (UID: \"891302e3-91c1-4b56-a169-950bb8587aa5\") " pod="openshift-cluster-node-tuning-operator/tuned-jf4g9"
Sep 17 23:10:20.664809 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:20.664747    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/1ae5e3b8-3cf3-4664-b502-8ab797838fb8-sys-fs\") pod \"aws-ebs-csi-driver-node-jph2d\" (UID: \"1ae5e3b8-3cf3-4664-b502-8ab797838fb8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jph2d"
Sep 17 23:10:20.664809 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:20.664781    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/123417a3-f884-4c2f-b7ac-837813412743-host-kubelet\") pod \"ovnkube-node-tcglq\" (UID: \"123417a3-f884-4c2f-b7ac-837813412743\") " pod="openshift-ovn-kubernetes/ovnkube-node-tcglq"
Sep 17 23:10:20.665037 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:20.664811    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/123417a3-f884-4c2f-b7ac-837813412743-host-run-netns\") pod \"ovnkube-node-tcglq\" (UID: \"123417a3-f884-4c2f-b7ac-837813412743\") " pod="openshift-ovn-kubernetes/ovnkube-node-tcglq"
Sep 17 23:10:20.665037 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:20.664834    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/123417a3-f884-4c2f-b7ac-837813412743-var-lib-openvswitch\") pod \"ovnkube-node-tcglq\" (UID: \"123417a3-f884-4c2f-b7ac-837813412743\") " pod="openshift-ovn-kubernetes/ovnkube-node-tcglq"
Sep 17 23:10:20.665037 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:20.664857    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/123417a3-f884-4c2f-b7ac-837813412743-host-run-ovn-kubernetes\") pod \"ovnkube-node-tcglq\" (UID: \"123417a3-f884-4c2f-b7ac-837813412743\") " pod="openshift-ovn-kubernetes/ovnkube-node-tcglq"
Sep 17 23:10:20.665037 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:20.664878    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/123417a3-f884-4c2f-b7ac-837813412743-host-cni-netd\") pod \"ovnkube-node-tcglq\" (UID: \"123417a3-f884-4c2f-b7ac-837813412743\") " pod="openshift-ovn-kubernetes/ovnkube-node-tcglq"
Sep 17 23:10:20.665037 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:20.664927    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/123417a3-f884-4c2f-b7ac-837813412743-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-tcglq\" (UID: \"123417a3-f884-4c2f-b7ac-837813412743\") " pod="openshift-ovn-kubernetes/ovnkube-node-tcglq"
Sep 17 23:10:20.665037 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:20.664964    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/123417a3-f884-4c2f-b7ac-837813412743-ovnkube-script-lib\") pod \"ovnkube-node-tcglq\" (UID: \"123417a3-f884-4c2f-b7ac-837813412743\") " pod="openshift-ovn-kubernetes/ovnkube-node-tcglq"
Sep 17 23:10:20.665037 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:20.664997    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/891302e3-91c1-4b56-a169-950bb8587aa5-etc-modprobe-d\") pod \"tuned-jf4g9\" (UID: \"891302e3-91c1-4b56-a169-950bb8587aa5\") " pod="openshift-cluster-node-tuning-operator/tuned-jf4g9"
Sep 17 23:10:20.665362 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:20.665055    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ktgmc\" (UniqueName: \"kubernetes.io/projected/fcc7ae4e-ace4-490f-a4e0-1c7c0ea22f2d-kube-api-access-ktgmc\") pod \"node-resolver-895t6\" (UID: \"fcc7ae4e-ace4-490f-a4e0-1c7c0ea22f2d\") " pod="openshift-dns/node-resolver-895t6"
Sep 17 23:10:20.665362 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:20.665083    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/04f96656-0988-4bea-a70f-79d1b8445201-serviceca\") pod \"node-ca-5spz7\" (UID: \"04f96656-0988-4bea-a70f-79d1b8445201\") " pod="openshift-image-registry/node-ca-5spz7"
Sep 17 23:10:20.665362 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:20.665108    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d672140f-ad59-4a68-9dde-80b62492af7f-metrics-certs\") pod \"network-metrics-daemon-4bbdp\" (UID: \"d672140f-ad59-4a68-9dde-80b62492af7f\") " pod="openshift-multus/network-metrics-daemon-4bbdp"
Sep 17 23:10:20.665362 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:20.665131    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l6p4l\" (UniqueName: \"kubernetes.io/projected/d672140f-ad59-4a68-9dde-80b62492af7f-kube-api-access-l6p4l\") pod \"network-metrics-daemon-4bbdp\" (UID: \"d672140f-ad59-4a68-9dde-80b62492af7f\") " pod="openshift-multus/network-metrics-daemon-4bbdp"
Sep 17 23:10:20.665362 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:20.665153    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/123417a3-f884-4c2f-b7ac-837813412743-run-openvswitch\") pod \"ovnkube-node-tcglq\" (UID: \"123417a3-f884-4c2f-b7ac-837813412743\") " pod="openshift-ovn-kubernetes/ovnkube-node-tcglq"
Sep 17 23:10:20.665362 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:20.665177    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/123417a3-f884-4c2f-b7ac-837813412743-env-overrides\") pod \"ovnkube-node-tcglq\" (UID: \"123417a3-f884-4c2f-b7ac-837813412743\") " pod="openshift-ovn-kubernetes/ovnkube-node-tcglq"
Sep 17 23:10:20.665362 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:20.665197    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/891302e3-91c1-4b56-a169-950bb8587aa5-etc-tuned\") pod \"tuned-jf4g9\" (UID: \"891302e3-91c1-4b56-a169-950bb8587aa5\") " pod="openshift-cluster-node-tuning-operator/tuned-jf4g9"
Sep 17 23:10:20.665362 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:20.665219    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f6e69d33-8d35-4bbd-b113-e16f0b87f42c-host-slash\") pod \"iptables-alerter-4bls7\" (UID: \"f6e69d33-8d35-4bbd-b113-e16f0b87f42c\") " pod="openshift-network-operator/iptables-alerter-4bls7"
Sep 17 23:10:20.665362 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:20.665243    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/fcc7ae4e-ace4-490f-a4e0-1c7c0ea22f2d-hosts-file\") pod \"node-resolver-895t6\" (UID: \"fcc7ae4e-ace4-490f-a4e0-1c7c0ea22f2d\") " pod="openshift-dns/node-resolver-895t6"
Sep 17 23:10:20.665362 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:20.665264    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fcc7ae4e-ace4-490f-a4e0-1c7c0ea22f2d-tmp-dir\") pod \"node-resolver-895t6\" (UID: \"fcc7ae4e-ace4-490f-a4e0-1c7c0ea22f2d\") " pod="openshift-dns/node-resolver-895t6"
Sep 17 23:10:20.665362 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:20.665285    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/1c71bd0e-1761-4157-82d0-a33701aece84-host-run-k8s-cni-cncf-io\") pod \"multus-9j2wk\" (UID: \"1c71bd0e-1761-4157-82d0-a33701aece84\") " pod="openshift-multus/multus-9j2wk"
Sep 17 23:10:20.665362 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:20.665306    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/123417a3-f884-4c2f-b7ac-837813412743-etc-openvswitch\") pod \"ovnkube-node-tcglq\" (UID: \"123417a3-f884-4c2f-b7ac-837813412743\") " pod="openshift-ovn-kubernetes/ovnkube-node-tcglq"
Sep 17 23:10:20.665362 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:20.665337    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/123417a3-f884-4c2f-b7ac-837813412743-ovn-node-metrics-cert\") pod \"ovnkube-node-tcglq\" (UID: \"123417a3-f884-4c2f-b7ac-837813412743\") " pod="openshift-ovn-kubernetes/ovnkube-node-tcglq"
Sep 17 23:10:20.665362 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:20.665368    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/891302e3-91c1-4b56-a169-950bb8587aa5-lib-modules\") pod \"tuned-jf4g9\" (UID: \"891302e3-91c1-4b56-a169-950bb8587aa5\") " pod="openshift-cluster-node-tuning-operator/tuned-jf4g9"
Sep 17 23:10:20.665990 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:20.665381    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f6e69d33-8d35-4bbd-b113-e16f0b87f42c-iptables-alerter-script\") pod \"iptables-alerter-4bls7\" (UID: \"f6e69d33-8d35-4bbd-b113-e16f0b87f42c\") " pod="openshift-network-operator/iptables-alerter-4bls7"
Sep 17 23:10:20.665990 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:20.665399    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1c71bd0e-1761-4157-82d0-a33701aece84-multus-conf-dir\") pod \"multus-9j2wk\" (UID: \"1c71bd0e-1761-4157-82d0-a33701aece84\") " pod="openshift-multus/multus-9j2wk"
Sep 17 23:10:20.665990 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:20.665414    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1c71bd0e-1761-4157-82d0-a33701aece84-multus-cni-dir\") pod \"multus-9j2wk\" (UID: \"1c71bd0e-1761-4157-82d0-a33701aece84\") " pod="openshift-multus/multus-9j2wk"
Sep 17 23:10:20.665990 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:20.665442    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1c71bd0e-1761-4157-82d0-a33701aece84-host-run-netns\") pod \"multus-9j2wk\" (UID: \"1c71bd0e-1761-4157-82d0-a33701aece84\") " pod="openshift-multus/multus-9j2wk"
Sep 17 23:10:20.665990 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:20.665479    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/04f96656-0988-4bea-a70f-79d1b8445201-host\") pod \"node-ca-5spz7\" (UID: \"04f96656-0988-4bea-a70f-79d1b8445201\") " pod="openshift-image-registry/node-ca-5spz7"
Sep 17 23:10:20.665990 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:20.665502    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/05765151-c799-4fe2-9f2e-cf4181e84ffa-agent-certs\") pod \"konnectivity-agent-g2vzq\" (UID: \"05765151-c799-4fe2-9f2e-cf4181e84ffa\") " pod="kube-system/konnectivity-agent-g2vzq"
Sep 17 23:10:20.665990 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:20.665525    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/891302e3-91c1-4b56-a169-950bb8587aa5-etc-kubernetes\") pod \"tuned-jf4g9\" (UID: \"891302e3-91c1-4b56-a169-950bb8587aa5\") " pod="openshift-cluster-node-tuning-operator/tuned-jf4g9"
Sep 17 23:10:20.665990 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:20.665562    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/891302e3-91c1-4b56-a169-950bb8587aa5-etc-systemd\") pod \"tuned-jf4g9\" (UID: \"891302e3-91c1-4b56-a169-950bb8587aa5\") " pod="openshift-cluster-node-tuning-operator/tuned-jf4g9"
Sep 17 23:10:20.665990 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:20.665596    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/fa08be4f-e950-4dee-a227-feb39c2ba7e0-kubelet-config\") pod \"global-pull-secret-syncer-24tsm\" (UID: \"fa08be4f-e950-4dee-a227-feb39c2ba7e0\") " pod="kube-system/global-pull-secret-syncer-24tsm"
Sep 17 23:10:20.665990 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:20.665616    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/891302e3-91c1-4b56-a169-950bb8587aa5-etc-sysconfig\") pod \"tuned-jf4g9\" (UID: \"891302e3-91c1-4b56-a169-950bb8587aa5\") " pod="openshift-cluster-node-tuning-operator/tuned-jf4g9"
Sep 17 23:10:20.665990 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:20.665654    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ea8e5a0a-14f1-4be6-84e1-33567e5d6032-cni-binary-copy\") pod \"multus-additional-cni-plugins-2gjg8\" (UID: \"ea8e5a0a-14f1-4be6-84e1-33567e5d6032\") " pod="openshift-multus/multus-additional-cni-plugins-2gjg8"
Sep 17 23:10:20.665990 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:20.665687    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1c71bd0e-1761-4157-82d0-a33701aece84-host-var-lib-cni-bin\") pod \"multus-9j2wk\" (UID: \"1c71bd0e-1761-4157-82d0-a33701aece84\") " pod="openshift-multus/multus-9j2wk"
Sep 17 23:10:20.665990 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:20.665708    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/891302e3-91c1-4b56-a169-950bb8587aa5-sys\") pod \"tuned-jf4g9\" (UID: \"891302e3-91c1-4b56-a169-950bb8587aa5\") " pod="openshift-cluster-node-tuning-operator/tuned-jf4g9"
Sep 17 23:10:20.665990 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:20.665746    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/891302e3-91c1-4b56-a169-950bb8587aa5-tmp\") pod \"tuned-jf4g9\" (UID: \"891302e3-91c1-4b56-a169-950bb8587aa5\") " pod="openshift-cluster-node-tuning-operator/tuned-jf4g9"
Sep 17 23:10:20.665990 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:20.665780    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bgsh5\" (UniqueName: \"kubernetes.io/projected/a2a26114-4c24-4dce-929c-3cb5caf17e6a-kube-api-access-bgsh5\") pod \"network-check-target-wgps9\" (UID: \"a2a26114-4c24-4dce-929c-3cb5caf17e6a\") " pod="openshift-network-diagnostics/network-check-target-wgps9"
Sep 17 23:10:20.665990 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:20.665812    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/ea8e5a0a-14f1-4be6-84e1-33567e5d6032-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-2gjg8\" (UID: \"ea8e5a0a-14f1-4be6-84e1-33567e5d6032\") " pod="openshift-multus/multus-additional-cni-plugins-2gjg8"
Sep 17 23:10:20.666735 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:20.665846    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1c71bd0e-1761-4157-82d0-a33701aece84-cni-binary-copy\") pod \"multus-9j2wk\" (UID: \"1c71bd0e-1761-4157-82d0-a33701aece84\") " pod="openshift-multus/multus-9j2wk"
Sep 17 23:10:20.666735 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:20.665892    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1c71bd0e-1761-4157-82d0-a33701aece84-host-var-lib-kubelet\") pod \"multus-9j2wk\" (UID: \"1c71bd0e-1761-4157-82d0-a33701aece84\") " pod="openshift-multus/multus-9j2wk"
Sep 17 23:10:20.666735 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:20.665915    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/123417a3-f884-4c2f-b7ac-837813412743-host-cni-bin\") pod \"ovnkube-node-tcglq\" (UID: \"123417a3-f884-4c2f-b7ac-837813412743\") " pod="openshift-ovn-kubernetes/ovnkube-node-tcglq"
Sep 17 23:10:20.666735 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:20.665942    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ea8e5a0a-14f1-4be6-84e1-33567e5d6032-cnibin\") pod \"multus-additional-cni-plugins-2gjg8\" (UID: \"ea8e5a0a-14f1-4be6-84e1-33567e5d6032\") " pod="openshift-multus/multus-additional-cni-plugins-2gjg8"
Sep 17 23:10:20.666735 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:20.665969    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/ea8e5a0a-14f1-4be6-84e1-33567e5d6032-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-2gjg8\" (UID: \"ea8e5a0a-14f1-4be6-84e1-33567e5d6032\") " pod="openshift-multus/multus-additional-cni-plugins-2gjg8"
Sep 17 23:10:20.666735 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:20.665996    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1c71bd0e-1761-4157-82d0-a33701aece84-system-cni-dir\") pod \"multus-9j2wk\" (UID: \"1c71bd0e-1761-4157-82d0-a33701aece84\") " pod="openshift-multus/multus-9j2wk"
Sep 17 23:10:20.666735 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:20.666041    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/123417a3-f884-4c2f-b7ac-837813412743-node-log\") pod \"ovnkube-node-tcglq\" (UID: \"123417a3-f884-4c2f-b7ac-837813412743\") " pod="openshift-ovn-kubernetes/ovnkube-node-tcglq"
Sep 17 23:10:20.666735 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:20.666066    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/891302e3-91c1-4b56-a169-950bb8587aa5-var-lib-kubelet\") pod \"tuned-jf4g9\" (UID: \"891302e3-91c1-4b56-a169-950bb8587aa5\") " pod="openshift-cluster-node-tuning-operator/tuned-jf4g9"
Sep 17 23:10:20.666735 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:20.666090    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7xl4v\" (UniqueName: \"kubernetes.io/projected/891302e3-91c1-4b56-a169-950bb8587aa5-kube-api-access-7xl4v\") pod \"tuned-jf4g9\" (UID: \"891302e3-91c1-4b56-a169-950bb8587aa5\") " pod="openshift-cluster-node-tuning-operator/tuned-jf4g9"
Sep 17 23:10:20.666735 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:20.666117    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wntvp\" (UniqueName: \"kubernetes.io/projected/1ae5e3b8-3cf3-4664-b502-8ab797838fb8-kube-api-access-wntvp\") pod \"aws-ebs-csi-driver-node-jph2d\" (UID: \"1ae5e3b8-3cf3-4664-b502-8ab797838fb8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jph2d"
Sep 17 23:10:20.666735 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:20.666143    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/1c71bd0e-1761-4157-82d0-a33701aece84-hostroot\") pod \"multus-9j2wk\" (UID: \"1c71bd0e-1761-4157-82d0-a33701aece84\") " pod="openshift-multus/multus-9j2wk"
Sep 17 23:10:20.666735 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:20.666173    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/fa08be4f-e950-4dee-a227-feb39c2ba7e0-dbus\") pod \"global-pull-secret-syncer-24tsm\" (UID: \"fa08be4f-e950-4dee-a227-feb39c2ba7e0\") " pod="kube-system/global-pull-secret-syncer-24tsm"
Sep 17 23:10:20.666735 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:20.666198    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/123417a3-f884-4c2f-b7ac-837813412743-run-systemd\") pod \"ovnkube-node-tcglq\" (UID: \"123417a3-f884-4c2f-b7ac-837813412743\") " pod="openshift-ovn-kubernetes/ovnkube-node-tcglq"
Sep 17 23:10:20.666735 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:20.666225    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zfbcm\" (UniqueName: \"kubernetes.io/projected/f6e69d33-8d35-4bbd-b113-e16f0b87f42c-kube-api-access-zfbcm\") pod \"iptables-alerter-4bls7\" (UID: \"f6e69d33-8d35-4bbd-b113-e16f0b87f42c\") " pod="openshift-network-operator/iptables-alerter-4bls7"
Sep 17 23:10:20.666735 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:20.666244    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/1c71bd0e-1761-4157-82d0-a33701aece84-multus-socket-dir-parent\") pod \"multus-9j2wk\" (UID: \"1c71bd0e-1761-4157-82d0-a33701aece84\") " pod="openshift-multus/multus-9j2wk"
Sep 17 23:10:20.666735 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:20.666259    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/1c71bd0e-1761-4157-82d0-a33701aece84-host-run-multus-certs\") pod \"multus-9j2wk\" (UID: \"1c71bd0e-1761-4157-82d0-a33701aece84\") " pod="openshift-multus/multus-9j2wk"
Sep 17 23:10:20.667416 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:20.666273    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/123417a3-f884-4c2f-b7ac-837813412743-systemd-units\") pod \"ovnkube-node-tcglq\" (UID: \"123417a3-f884-4c2f-b7ac-837813412743\") " pod="openshift-ovn-kubernetes/ovnkube-node-tcglq"
Sep 17 23:10:20.667416 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:20.666297    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/1ae5e3b8-3cf3-4664-b502-8ab797838fb8-kubelet-dir\") pod \"aws-ebs-csi-driver-node-jph2d\" (UID: \"1ae5e3b8-3cf3-4664-b502-8ab797838fb8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jph2d"
Sep 17 23:10:20.667416 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:20.666314    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/1ae5e3b8-3cf3-4664-b502-8ab797838fb8-etc-selinux\") pod \"aws-ebs-csi-driver-node-jph2d\" (UID: \"1ae5e3b8-3cf3-4664-b502-8ab797838fb8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jph2d"
Sep 17 23:10:20.667416 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:20.666332    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5lhqm\" (UniqueName: \"kubernetes.io/projected/ea8e5a0a-14f1-4be6-84e1-33567e5d6032-kube-api-access-5lhqm\") pod \"multus-additional-cni-plugins-2gjg8\" (UID: \"ea8e5a0a-14f1-4be6-84e1-33567e5d6032\") " pod="openshift-multus/multus-additional-cni-plugins-2gjg8"
Sep 17 23:10:20.667416 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:20.666347    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fa08be4f-e950-4dee-a227-feb39c2ba7e0-original-pull-secret\") pod \"global-pull-secret-syncer-24tsm\" (UID: \"fa08be4f-e950-4dee-a227-feb39c2ba7e0\") " pod="kube-system/global-pull-secret-syncer-24tsm"
Sep 17 23:10:20.667416 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:20.666366    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/123417a3-f884-4c2f-b7ac-837813412743-log-socket\") pod \"ovnkube-node-tcglq\" (UID: \"123417a3-f884-4c2f-b7ac-837813412743\") " pod="openshift-ovn-kubernetes/ovnkube-node-tcglq"
Sep 17 23:10:20.667416 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:20.666399    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/123417a3-f884-4c2f-b7ac-837813412743-ovnkube-config\") pod \"ovnkube-node-tcglq\" (UID: \"123417a3-f884-4c2f-b7ac-837813412743\") " pod="openshift-ovn-kubernetes/ovnkube-node-tcglq"
Sep 17 23:10:20.667416 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:20.666420    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x7h7k\" (UniqueName: \"kubernetes.io/projected/123417a3-f884-4c2f-b7ac-837813412743-kube-api-access-x7h7k\") pod \"ovnkube-node-tcglq\" (UID: \"123417a3-f884-4c2f-b7ac-837813412743\") " pod="openshift-ovn-kubernetes/ovnkube-node-tcglq"
Sep 17 23:10:20.667416 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:20.666439    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/891302e3-91c1-4b56-a169-950bb8587aa5-etc-sysctl-conf\") pod \"tuned-jf4g9\" (UID: \"891302e3-91c1-4b56-a169-950bb8587aa5\") " pod="openshift-cluster-node-tuning-operator/tuned-jf4g9"
Sep 17 23:10:20.667416 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:20.666484    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ea8e5a0a-14f1-4be6-84e1-33567e5d6032-tuning-conf-dir\") pod \"multus-additional-cni-plugins-2gjg8\" (UID: \"ea8e5a0a-14f1-4be6-84e1-33567e5d6032\") " pod="openshift-multus/multus-additional-cni-plugins-2gjg8"
Sep 17 23:10:20.667416 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:20.666508    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1c71bd0e-1761-4157-82d0-a33701aece84-cnibin\") pod \"multus-9j2wk\" (UID: \"1c71bd0e-1761-4157-82d0-a33701aece84\") " pod="openshift-multus/multus-9j2wk"
Sep 17 23:10:20.667416 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:20.666535    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/1c71bd0e-1761-4157-82d0-a33701aece84-multus-daemon-config\") pod \"multus-9j2wk\" (UID: \"1c71bd0e-1761-4157-82d0-a33701aece84\") " pod="openshift-multus/multus-9j2wk"
Sep 17 23:10:20.667416 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:20.666554    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cwfm4\" (UniqueName: \"kubernetes.io/projected/04f96656-0988-4bea-a70f-79d1b8445201-kube-api-access-cwfm4\") pod \"node-ca-5spz7\" (UID: \"04f96656-0988-4bea-a70f-79d1b8445201\") " pod="openshift-image-registry/node-ca-5spz7"
Sep 17 23:10:20.667416 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:20.666568    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/123417a3-f884-4c2f-b7ac-837813412743-host-slash\") pod \"ovnkube-node-tcglq\" (UID: \"123417a3-f884-4c2f-b7ac-837813412743\") " pod="openshift-ovn-kubernetes/ovnkube-node-tcglq"
Sep 17 23:10:20.667416 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:20.666583    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/1ae5e3b8-3cf3-4664-b502-8ab797838fb8-registration-dir\") pod \"aws-ebs-csi-driver-node-jph2d\" (UID: \"1ae5e3b8-3cf3-4664-b502-8ab797838fb8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jph2d"
Sep 17 23:10:20.667416 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:20.666597    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/1c71bd0e-1761-4157-82d0-a33701aece84-host-var-lib-cni-multus\") pod \"multus-9j2wk\" (UID: \"1c71bd0e-1761-4157-82d0-a33701aece84\") " pod="openshift-multus/multus-9j2wk"
Sep 17 23:10:20.667899 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:20.666630    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1c71bd0e-1761-4157-82d0-a33701aece84-etc-kubernetes\") pod \"multus-9j2wk\" (UID: \"1c71bd0e-1761-4157-82d0-a33701aece84\") " pod="openshift-multus/multus-9j2wk"
Sep 17 23:10:20.667899 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:20.666644    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zmhgw\" (UniqueName: \"kubernetes.io/projected/1c71bd0e-1761-4157-82d0-a33701aece84-kube-api-access-zmhgw\") pod \"multus-9j2wk\" (UID: \"1c71bd0e-1761-4157-82d0-a33701aece84\") " pod="openshift-multus/multus-9j2wk"
Sep 17 23:10:20.667899 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:20.666666    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/891302e3-91c1-4b56-a169-950bb8587aa5-host\") pod \"tuned-jf4g9\" (UID: \"891302e3-91c1-4b56-a169-950bb8587aa5\") " pod="openshift-cluster-node-tuning-operator/tuned-jf4g9"
Sep 17 23:10:20.674299 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:20.674132    2584 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 17 23:10:20.709111 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:20.709093    2584 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-qrlg4"
Sep 17 23:10:20.722394 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:20.722374    2584 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-qrlg4"
Sep 17 23:10:20.762846 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:20.762798    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-4.ec2.internal" event={"ID":"68edadcfa29cc8be7e22dcaea45c88a8","Type":"ContainerStarted","Data":"3e6291a26948715d6e31e67de35259f57a966ca410921115cd8974aa88bd674c"}
Sep 17 23:10:20.764147 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:20.764121    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-4.ec2.internal" event={"ID":"3771e4c220b5a223d0f484f25cdc29e8","Type":"ContainerStarted","Data":"4660d0d9e27e123f5aa0bc38fa39e98770d7949ae48447f34adbcc3aa2817032"}
Sep 17 23:10:20.766919 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:20.766894    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/891302e3-91c1-4b56-a169-950bb8587aa5-etc-kubernetes\") pod \"tuned-jf4g9\" (UID: \"891302e3-91c1-4b56-a169-950bb8587aa5\") " pod="openshift-cluster-node-tuning-operator/tuned-jf4g9"
Sep 17 23:10:20.767054 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:20.766931    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/891302e3-91c1-4b56-a169-950bb8587aa5-etc-systemd\") pod \"tuned-jf4g9\" (UID: \"891302e3-91c1-4b56-a169-950bb8587aa5\") " pod="openshift-cluster-node-tuning-operator/tuned-jf4g9"
Sep 17 23:10:20.767054 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:20.766949    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/fa08be4f-e950-4dee-a227-feb39c2ba7e0-kubelet-config\") pod \"global-pull-secret-syncer-24tsm\" (UID: \"fa08be4f-e950-4dee-a227-feb39c2ba7e0\") " pod="kube-system/global-pull-secret-syncer-24tsm"
Sep 17 23:10:20.767054 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:20.766978    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/891302e3-91c1-4b56-a169-950bb8587aa5-etc-sysconfig\") pod \"tuned-jf4g9\" (UID: \"891302e3-91c1-4b56-a169-950bb8587aa5\") " pod="openshift-cluster-node-tuning-operator/tuned-jf4g9"
Sep 17 23:10:20.767054 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:20.766992    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ea8e5a0a-14f1-4be6-84e1-33567e5d6032-cni-binary-copy\") pod \"multus-additional-cni-plugins-2gjg8\" (UID: \"ea8e5a0a-14f1-4be6-84e1-33567e5d6032\") " pod="openshift-multus/multus-additional-cni-plugins-2gjg8"
Sep 17 23:10:20.767054 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:20.767050    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/891302e3-91c1-4b56-a169-950bb8587aa5-etc-kubernetes\") pod \"tuned-jf4g9\" (UID: \"891302e3-91c1-4b56-a169-950bb8587aa5\") " pod="openshift-cluster-node-tuning-operator/tuned-jf4g9"
Sep 17 23:10:20.767308 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:20.767050    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/891302e3-91c1-4b56-a169-950bb8587aa5-etc-systemd\") pod \"tuned-jf4g9\" (UID: \"891302e3-91c1-4b56-a169-950bb8587aa5\") " pod="openshift-cluster-node-tuning-operator/tuned-jf4g9"
Sep 17 23:10:20.767308 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:20.767077    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/fa08be4f-e950-4dee-a227-feb39c2ba7e0-kubelet-config\") pod \"global-pull-secret-syncer-24tsm\" (UID: \"fa08be4f-e950-4dee-a227-feb39c2ba7e0\") " pod="kube-system/global-pull-secret-syncer-24tsm"
Sep 17 23:10:20.767308 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:20.767095    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/891302e3-91c1-4b56-a169-950bb8587aa5-etc-sysconfig\") pod \"tuned-jf4g9\" (UID: \"891302e3-91c1-4b56-a169-950bb8587aa5\") " pod="openshift-cluster-node-tuning-operator/tuned-jf4g9"
Sep 17 23:10:20.767308 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:20.767103    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1c71bd0e-1761-4157-82d0-a33701aece84-host-var-lib-cni-bin\") pod \"multus-9j2wk\" (UID: \"1c71bd0e-1761-4157-82d0-a33701aece84\") " pod="openshift-multus/multus-9j2wk"
Sep 17 23:10:20.767308 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:20.767129    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/891302e3-91c1-4b56-a169-950bb8587aa5-sys\") pod \"tuned-jf4g9\" (UID: \"891302e3-91c1-4b56-a169-950bb8587aa5\") " pod="openshift-cluster-node-tuning-operator/tuned-jf4g9"
Sep 17 23:10:20.767308 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:20.767150    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/891302e3-91c1-4b56-a169-950bb8587aa5-tmp\") pod \"tuned-jf4g9\" (UID: \"891302e3-91c1-4b56-a169-950bb8587aa5\") " pod="openshift-cluster-node-tuning-operator/tuned-jf4g9"
Sep 17 23:10:20.767308 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:20.767170    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bgsh5\" (UniqueName: \"kubernetes.io/projected/a2a26114-4c24-4dce-929c-3cb5caf17e6a-kube-api-access-bgsh5\") pod \"network-check-target-wgps9\" (UID: \"a2a26114-4c24-4dce-929c-3cb5caf17e6a\") " pod="openshift-network-diagnostics/network-check-target-wgps9"
Sep 17 23:10:20.767308 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:20.767194    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/ea8e5a0a-14f1-4be6-84e1-33567e5d6032-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-2gjg8\" (UID: \"ea8e5a0a-14f1-4be6-84e1-33567e5d6032\") " pod="openshift-multus/multus-additional-cni-plugins-2gjg8"
Sep 17 23:10:20.767308 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:20.767199    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1c71bd0e-1761-4157-82d0-a33701aece84-host-var-lib-cni-bin\") pod \"multus-9j2wk\" (UID: \"1c71bd0e-1761-4157-82d0-a33701aece84\") " pod="openshift-multus/multus-9j2wk"
Sep 17 23:10:20.767308 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:20.767204    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/891302e3-91c1-4b56-a169-950bb8587aa5-sys\") pod \"tuned-jf4g9\" (UID: \"891302e3-91c1-4b56-a169-950bb8587aa5\") " pod="openshift-cluster-node-tuning-operator/tuned-jf4g9"
Sep 17 23:10:20.767308 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:20.767217    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1c71bd0e-1761-4157-82d0-a33701aece84-cni-binary-copy\") pod \"multus-9j2wk\" (UID: \"1c71bd0e-1761-4157-82d0-a33701aece84\") " pod="openshift-multus/multus-9j2wk"
Sep 17 23:10:20.767308 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:20.767239    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1c71bd0e-1761-4157-82d0-a33701aece84-host-var-lib-kubelet\") pod \"multus-9j2wk\" (UID: \"1c71bd0e-1761-4157-82d0-a33701aece84\") " pod="openshift-multus/multus-9j2wk"
Sep 17 23:10:20.767308 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:20.767260    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/123417a3-f884-4c2f-b7ac-837813412743-host-cni-bin\") pod \"ovnkube-node-tcglq\" (UID: \"123417a3-f884-4c2f-b7ac-837813412743\") " pod="openshift-ovn-kubernetes/ovnkube-node-tcglq"
Sep 17 23:10:20.767308 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:20.767281    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ea8e5a0a-14f1-4be6-84e1-33567e5d6032-cnibin\") pod \"multus-additional-cni-plugins-2gjg8\" (UID: \"ea8e5a0a-14f1-4be6-84e1-33567e5d6032\") " pod="openshift-multus/multus-additional-cni-plugins-2gjg8"
Sep 17 23:10:20.767308 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:20.767305    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/ea8e5a0a-14f1-4be6-84e1-33567e5d6032-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-2gjg8\" (UID: \"ea8e5a0a-14f1-4be6-84e1-33567e5d6032\") " pod="openshift-multus/multus-additional-cni-plugins-2gjg8"
Sep 17 23:10:20.768080 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:20.767327    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1c71bd0e-1761-4157-82d0-a33701aece84-system-cni-dir\") pod \"multus-9j2wk\" (UID: \"1c71bd0e-1761-4157-82d0-a33701aece84\") " pod="openshift-multus/multus-9j2wk"
Sep 17 23:10:20.768080 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:20.767355    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/123417a3-f884-4c2f-b7ac-837813412743-node-log\") pod \"ovnkube-node-tcglq\" (UID: \"123417a3-f884-4c2f-b7ac-837813412743\") " pod="openshift-ovn-kubernetes/ovnkube-node-tcglq"
Sep 17 23:10:20.768080 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:20.767375    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/891302e3-91c1-4b56-a169-950bb8587aa5-var-lib-kubelet\") pod \"tuned-jf4g9\" (UID: \"891302e3-91c1-4b56-a169-950bb8587aa5\") " pod="openshift-cluster-node-tuning-operator/tuned-jf4g9"
Sep 17 23:10:20.768080 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:20.767396    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7xl4v\" (UniqueName: \"kubernetes.io/projected/891302e3-91c1-4b56-a169-950bb8587aa5-kube-api-access-7xl4v\") pod \"tuned-jf4g9\" (UID: \"891302e3-91c1-4b56-a169-950bb8587aa5\") " pod="openshift-cluster-node-tuning-operator/tuned-jf4g9"
Sep 17 23:10:20.768080 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:20.767434    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ea8e5a0a-14f1-4be6-84e1-33567e5d6032-cnibin\") pod \"multus-additional-cni-plugins-2gjg8\" (UID: \"ea8e5a0a-14f1-4be6-84e1-33567e5d6032\") " pod="openshift-multus/multus-additional-cni-plugins-2gjg8"
Sep 17 23:10:20.768080 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:20.767439    2584 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 17 23:10:20.768080 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:20.767458    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wntvp\" (UniqueName: \"kubernetes.io/projected/1ae5e3b8-3cf3-4664-b502-8ab797838fb8-kube-api-access-wntvp\") pod \"aws-ebs-csi-driver-node-jph2d\" (UID: \"1ae5e3b8-3cf3-4664-b502-8ab797838fb8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jph2d"
Sep 17 23:10:20.768080 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:20.767482    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/1c71bd0e-1761-4157-82d0-a33701aece84-hostroot\") pod \"multus-9j2wk\" (UID: \"1c71bd0e-1761-4157-82d0-a33701aece84\") " pod="openshift-multus/multus-9j2wk"
Sep 17 23:10:20.768080 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:20.767504    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/fa08be4f-e950-4dee-a227-feb39c2ba7e0-dbus\") pod \"global-pull-secret-syncer-24tsm\" (UID: \"fa08be4f-e950-4dee-a227-feb39c2ba7e0\") " pod="kube-system/global-pull-secret-syncer-24tsm"
Sep 17 23:10:20.768080 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:20.767523    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ea8e5a0a-14f1-4be6-84e1-33567e5d6032-cni-binary-copy\") pod \"multus-additional-cni-plugins-2gjg8\" (UID: \"ea8e5a0a-14f1-4be6-84e1-33567e5d6032\") " pod="openshift-multus/multus-additional-cni-plugins-2gjg8"
Sep 17 23:10:20.768080 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:20.767557    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/123417a3-f884-4c2f-b7ac-837813412743-run-systemd\") pod \"ovnkube-node-tcglq\" (UID: \"123417a3-f884-4c2f-b7ac-837813412743\") " pod="openshift-ovn-kubernetes/ovnkube-node-tcglq"
Sep 17 23:10:20.768080 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:20.767527    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/123417a3-f884-4c2f-b7ac-837813412743-run-systemd\") pod \"ovnkube-node-tcglq\" (UID: \"123417a3-f884-4c2f-b7ac-837813412743\") " pod="openshift-ovn-kubernetes/ovnkube-node-tcglq"
Sep 17 23:10:20.768080 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:20.767600    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1c71bd0e-1761-4157-82d0-a33701aece84-host-var-lib-kubelet\") pod \"multus-9j2wk\" (UID: \"1c71bd0e-1761-4157-82d0-a33701aece84\") " pod="openshift-multus/multus-9j2wk"
Sep 17 23:10:20.768080 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:20.767608    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zfbcm\" (UniqueName: \"kubernetes.io/projected/f6e69d33-8d35-4bbd-b113-e16f0b87f42c-kube-api-access-zfbcm\") pod \"iptables-alerter-4bls7\" (UID: \"f6e69d33-8d35-4bbd-b113-e16f0b87f42c\") " pod="openshift-network-operator/iptables-alerter-4bls7"
Sep 17 23:10:20.768080 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:20.767626    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1c71bd0e-1761-4157-82d0-a33701aece84-system-cni-dir\") pod \"multus-9j2wk\" (UID: \"1c71bd0e-1761-4157-82d0-a33701aece84\") " pod="openshift-multus/multus-9j2wk"
Sep 17 23:10:20.768771 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:20.768348    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/1c71bd0e-1761-4157-82d0-a33701aece84-multus-socket-dir-parent\") pod \"multus-9j2wk\" (UID: \"1c71bd0e-1761-4157-82d0-a33701aece84\") " pod="openshift-multus/multus-9j2wk"
Sep 17 23:10:20.768771 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:20.768408    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/1c71bd0e-1761-4157-82d0-a33701aece84-host-run-multus-certs\") pod \"multus-9j2wk\" (UID: \"1c71bd0e-1761-4157-82d0-a33701aece84\") " pod="openshift-multus/multus-9j2wk"
Sep 17 23:10:20.768771 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:20.768433    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/123417a3-f884-4c2f-b7ac-837813412743-systemd-units\") pod \"ovnkube-node-tcglq\" (UID: \"123417a3-f884-4c2f-b7ac-837813412743\") " pod="openshift-ovn-kubernetes/ovnkube-node-tcglq"
Sep 17 23:10:20.768771 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:20.768466    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/1ae5e3b8-3cf3-4664-b502-8ab797838fb8-kubelet-dir\") pod \"aws-ebs-csi-driver-node-jph2d\" (UID: \"1ae5e3b8-3cf3-4664-b502-8ab797838fb8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jph2d"
Sep 17 23:10:20.768771 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:20.768495    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/1ae5e3b8-3cf3-4664-b502-8ab797838fb8-etc-selinux\") pod \"aws-ebs-csi-driver-node-jph2d\" (UID: \"1ae5e3b8-3cf3-4664-b502-8ab797838fb8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jph2d"
Sep 17 23:10:20.768771 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:20.768525    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5lhqm\" (UniqueName: \"kubernetes.io/projected/ea8e5a0a-14f1-4be6-84e1-33567e5d6032-kube-api-access-5lhqm\") pod \"multus-additional-cni-plugins-2gjg8\" (UID: \"ea8e5a0a-14f1-4be6-84e1-33567e5d6032\") " pod="openshift-multus/multus-additional-cni-plugins-2gjg8"
Sep 17 23:10:20.768771 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:20.768553    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fa08be4f-e950-4dee-a227-feb39c2ba7e0-original-pull-secret\") pod \"global-pull-secret-syncer-24tsm\" (UID: \"fa08be4f-e950-4dee-a227-feb39c2ba7e0\") " pod="kube-system/global-pull-secret-syncer-24tsm"
Sep 17 23:10:20.768771 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:20.768582    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/123417a3-f884-4c2f-b7ac-837813412743-log-socket\") pod \"ovnkube-node-tcglq\" (UID: \"123417a3-f884-4c2f-b7ac-837813412743\") " pod="openshift-ovn-kubernetes/ovnkube-node-tcglq"
Sep 17 23:10:20.768771 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:20.768611    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/123417a3-f884-4c2f-b7ac-837813412743-ovnkube-config\") pod \"ovnkube-node-tcglq\" (UID: \"123417a3-f884-4c2f-b7ac-837813412743\") " pod="openshift-ovn-kubernetes/ovnkube-node-tcglq"
Sep 17 23:10:20.768771 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:20.768641    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x7h7k\" (UniqueName: \"kubernetes.io/projected/123417a3-f884-4c2f-b7ac-837813412743-kube-api-access-x7h7k\") pod \"ovnkube-node-tcglq\" (UID: \"123417a3-f884-4c2f-b7ac-837813412743\") " pod="openshift-ovn-kubernetes/ovnkube-node-tcglq"
Sep 17 23:10:20.768771 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:20.768669    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/891302e3-91c1-4b56-a169-950bb8587aa5-etc-sysctl-conf\") pod \"tuned-jf4g9\" (UID: \"891302e3-91c1-4b56-a169-950bb8587aa5\") " pod="openshift-cluster-node-tuning-operator/tuned-jf4g9"
Sep 17 23:10:20.768771 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:20.768698    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ea8e5a0a-14f1-4be6-84e1-33567e5d6032-tuning-conf-dir\") pod \"multus-additional-cni-plugins-2gjg8\" (UID: \"ea8e5a0a-14f1-4be6-84e1-33567e5d6032\") " pod="openshift-multus/multus-additional-cni-plugins-2gjg8"
Sep 17 23:10:20.768771 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:20.768721    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1c71bd0e-1761-4157-82d0-a33701aece84-cnibin\") pod \"multus-9j2wk\" (UID: \"1c71bd0e-1761-4157-82d0-a33701aece84\") " pod="openshift-multus/multus-9j2wk"
Sep 17 23:10:20.768771 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:20.768749    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/1c71bd0e-1761-4157-82d0-a33701aece84-multus-daemon-config\") pod \"multus-9j2wk\" (UID: \"1c71bd0e-1761-4157-82d0-a33701aece84\") " pod="openshift-multus/multus-9j2wk"
Sep 17 23:10:20.768771 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:20.768777    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cwfm4\" (UniqueName: \"kubernetes.io/projected/04f96656-0988-4bea-a70f-79d1b8445201-kube-api-access-cwfm4\") pod \"node-ca-5spz7\" (UID: \"04f96656-0988-4bea-a70f-79d1b8445201\") " pod="openshift-image-registry/node-ca-5spz7"
Sep 17 23:10:20.769364 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:20.768805    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/123417a3-f884-4c2f-b7ac-837813412743-host-slash\") pod \"ovnkube-node-tcglq\" (UID: \"123417a3-f884-4c2f-b7ac-837813412743\") " pod="openshift-ovn-kubernetes/ovnkube-node-tcglq"
Sep 17 23:10:20.769364 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:20.768860    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/123417a3-f884-4c2f-b7ac-837813412743-host-slash\") pod \"ovnkube-node-tcglq\" (UID: \"123417a3-f884-4c2f-b7ac-837813412743\") " pod="openshift-ovn-kubernetes/ovnkube-node-tcglq"
Sep 17 23:10:20.769364 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:20.769180    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/123417a3-f884-4c2f-b7ac-837813412743-node-log\") pod \"ovnkube-node-tcglq\" (UID: \"123417a3-f884-4c2f-b7ac-837813412743\") " pod="openshift-ovn-kubernetes/ovnkube-node-tcglq"
Sep 17 23:10:20.769562 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:20.769470    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/891302e3-91c1-4b56-a169-950bb8587aa5-var-lib-kubelet\") pod \"tuned-jf4g9\" (UID: \"891302e3-91c1-4b56-a169-950bb8587aa5\") " pod="openshift-cluster-node-tuning-operator/tuned-jf4g9"
Sep 17 23:10:20.769915 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:20.769892    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/1ae5e3b8-3cf3-4664-b502-8ab797838fb8-registration-dir\") pod \"aws-ebs-csi-driver-node-jph2d\" (UID: \"1ae5e3b8-3cf3-4664-b502-8ab797838fb8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jph2d"
Sep 17 23:10:20.769966 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:20.769939    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/1c71bd0e-1761-4157-82d0-a33701aece84-host-var-lib-cni-multus\") pod \"multus-9j2wk\" (UID: \"1c71bd0e-1761-4157-82d0-a33701aece84\") " pod="openshift-multus/multus-9j2wk"
Sep 17 23:10:20.769997 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:20.769967    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1c71bd0e-1761-4157-82d0-a33701aece84-etc-kubernetes\") pod \"multus-9j2wk\" (UID: \"1c71bd0e-1761-4157-82d0-a33701aece84\") " pod="openshift-multus/multus-9j2wk"
Sep 17 23:10:20.770057 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:20.770023    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zmhgw\" (UniqueName: \"kubernetes.io/projected/1c71bd0e-1761-4157-82d0-a33701aece84-kube-api-access-zmhgw\") pod \"multus-9j2wk\" (UID: \"1c71bd0e-1761-4157-82d0-a33701aece84\") " pod="openshift-multus/multus-9j2wk"
Sep 17 23:10:20.770057 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:20.770044    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/1ae5e3b8-3cf3-4664-b502-8ab797838fb8-registration-dir\") pod \"aws-ebs-csi-driver-node-jph2d\" (UID: \"1ae5e3b8-3cf3-4664-b502-8ab797838fb8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jph2d"
Sep 17 23:10:20.770130 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:20.770051    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/891302e3-91c1-4b56-a169-950bb8587aa5-host\") pod \"tuned-jf4g9\" (UID: \"891302e3-91c1-4b56-a169-950bb8587aa5\") " pod="openshift-cluster-node-tuning-operator/tuned-jf4g9"
Sep 17 23:10:20.770219 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:20.770203    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/891302e3-91c1-4b56-a169-950bb8587aa5-run\") pod \"tuned-jf4g9\" (UID: \"891302e3-91c1-4b56-a169-950bb8587aa5\") " pod="openshift-cluster-node-tuning-operator/tuned-jf4g9"
Sep 17 23:10:20.770254 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:20.770236    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/1ae5e3b8-3cf3-4664-b502-8ab797838fb8-socket-dir\") pod \"aws-ebs-csi-driver-node-jph2d\" (UID: \"1ae5e3b8-3cf3-4664-b502-8ab797838fb8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jph2d"
Sep 17 23:10:20.770286 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:20.770266    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/1ae5e3b8-3cf3-4664-b502-8ab797838fb8-device-dir\") pod \"aws-ebs-csi-driver-node-jph2d\" (UID: \"1ae5e3b8-3cf3-4664-b502-8ab797838fb8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jph2d"
Sep 17 23:10:20.770286 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:20.770269    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/1c71bd0e-1761-4157-82d0-a33701aece84-host-run-multus-certs\") pod \"multus-9j2wk\" (UID: \"1c71bd0e-1761-4157-82d0-a33701aece84\") " pod="openshift-multus/multus-9j2wk"
Sep 17 23:10:20.770350 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:20.770313    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/123417a3-f884-4c2f-b7ac-837813412743-systemd-units\") pod \"ovnkube-node-tcglq\" (UID: \"123417a3-f884-4c2f-b7ac-837813412743\") " pod="openshift-ovn-kubernetes/ovnkube-node-tcglq"
Sep 17 23:10:20.770350 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:20.770301    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/1c71bd0e-1761-4157-82d0-a33701aece84-host-var-lib-cni-multus\") pod \"multus-9j2wk\" (UID: \"1c71bd0e-1761-4157-82d0-a33701aece84\") " pod="openshift-multus/multus-9j2wk"
Sep 17 23:10:20.770350 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:20.770323    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/1ae5e3b8-3cf3-4664-b502-8ab797838fb8-device-dir\") pod \"aws-ebs-csi-driver-node-jph2d\" (UID: \"1ae5e3b8-3cf3-4664-b502-8ab797838fb8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jph2d"
Sep 17 23:10:20.770437 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:20.770366    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ea8e5a0a-14f1-4be6-84e1-33567e5d6032-system-cni-dir\") pod \"multus-additional-cni-plugins-2gjg8\" (UID: \"ea8e5a0a-14f1-4be6-84e1-33567e5d6032\") " pod="openshift-multus/multus-additional-cni-plugins-2gjg8"
Sep 17 23:10:20.770437 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:20.770400    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/05765151-c799-4fe2-9f2e-cf4181e84ffa-konnectivity-ca\") pod \"konnectivity-agent-g2vzq\" (UID: \"05765151-c799-4fe2-9f2e-cf4181e84ffa\") " pod="kube-system/konnectivity-agent-g2vzq"
Sep 17 23:10:20.770437 ip-10-0-142-4 kubenswrapper[2584]: E0917 23:10:20.770405    2584 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 23:10:20.770520 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:20.770456    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/1ae5e3b8-3cf3-4664-b502-8ab797838fb8-kubelet-dir\") pod \"aws-ebs-csi-driver-node-jph2d\" (UID: \"1ae5e3b8-3cf3-4664-b502-8ab797838fb8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jph2d"
Sep 17 23:10:20.770520 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:20.770502    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/891302e3-91c1-4b56-a169-950bb8587aa5-run\") pod \"tuned-jf4g9\" (UID: \"891302e3-91c1-4b56-a169-950bb8587aa5\") " pod="openshift-cluster-node-tuning-operator/tuned-jf4g9"
Sep 17 23:10:20.770580 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:20.770519    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ea8e5a0a-14f1-4be6-84e1-33567e5d6032-system-cni-dir\") pod \"multus-additional-cni-plugins-2gjg8\" (UID: \"ea8e5a0a-14f1-4be6-84e1-33567e5d6032\") " pod="openshift-multus/multus-additional-cni-plugins-2gjg8"
Sep 17 23:10:20.770580 ip-10-0-142-4 kubenswrapper[2584]: E0917 23:10:20.770560    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fa08be4f-e950-4dee-a227-feb39c2ba7e0-original-pull-secret podName:fa08be4f-e950-4dee-a227-feb39c2ba7e0 nodeName:}" failed. No retries permitted until 2026-09-17 23:10:21.270520006 +0000 UTC m=+4.636179924 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fa08be4f-e950-4dee-a227-feb39c2ba7e0-original-pull-secret") pod "global-pull-secret-syncer-24tsm" (UID: "fa08be4f-e950-4dee-a227-feb39c2ba7e0") : object "kube-system"/"original-pull-secret" not registered
Sep 17 23:10:20.770648 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:20.770601    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/1ae5e3b8-3cf3-4664-b502-8ab797838fb8-socket-dir\") pod \"aws-ebs-csi-driver-node-jph2d\" (UID: \"1ae5e3b8-3cf3-4664-b502-8ab797838fb8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jph2d"
Sep 17 23:10:20.770683 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:20.770644    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/891302e3-91c1-4b56-a169-950bb8587aa5-host\") pod \"tuned-jf4g9\" (UID: \"891302e3-91c1-4b56-a169-950bb8587aa5\") " pod="openshift-cluster-node-tuning-operator/tuned-jf4g9"
Sep 17 23:10:20.770733 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:20.770717    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/1ae5e3b8-3cf3-4664-b502-8ab797838fb8-etc-selinux\") pod \"aws-ebs-csi-driver-node-jph2d\" (UID: \"1ae5e3b8-3cf3-4664-b502-8ab797838fb8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jph2d"
Sep 17 23:10:20.770768 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:20.770757    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/123417a3-f884-4c2f-b7ac-837813412743-log-socket\") pod \"ovnkube-node-tcglq\" (UID: \"123417a3-f884-4c2f-b7ac-837813412743\") " pod="openshift-ovn-kubernetes/ovnkube-node-tcglq"
Sep 17 23:10:20.771115 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:20.771085    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/ea8e5a0a-14f1-4be6-84e1-33567e5d6032-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-2gjg8\" (UID: \"ea8e5a0a-14f1-4be6-84e1-33567e5d6032\") " pod="openshift-multus/multus-additional-cni-plugins-2gjg8"
Sep 17 23:10:20.771195 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:20.770233    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/1c71bd0e-1761-4157-82d0-a33701aece84-multus-socket-dir-parent\") pod \"multus-9j2wk\" (UID: \"1c71bd0e-1761-4157-82d0-a33701aece84\") " pod="openshift-multus/multus-9j2wk"
Sep 17 23:10:20.771195 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:20.771149    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/123417a3-f884-4c2f-b7ac-837813412743-host-cni-bin\") pod \"ovnkube-node-tcglq\" (UID: \"123417a3-f884-4c2f-b7ac-837813412743\") " pod="openshift-ovn-kubernetes/ovnkube-node-tcglq"
Sep 17 23:10:20.771195 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:20.771170    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/123417a3-f884-4c2f-b7ac-837813412743-run-ovn\") pod \"ovnkube-node-tcglq\" (UID: \"123417a3-f884-4c2f-b7ac-837813412743\") " pod="openshift-ovn-kubernetes/ovnkube-node-tcglq"
Sep 17 23:10:20.771368 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:20.771212    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/891302e3-91c1-4b56-a169-950bb8587aa5-etc-sysctl-d\") pod \"tuned-jf4g9\" (UID: \"891302e3-91c1-4b56-a169-950bb8587aa5\") " pod="openshift-cluster-node-tuning-operator/tuned-jf4g9"
Sep 17 23:10:20.771368 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:20.771240    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/1ae5e3b8-3cf3-4664-b502-8ab797838fb8-sys-fs\") pod \"aws-ebs-csi-driver-node-jph2d\" (UID: \"1ae5e3b8-3cf3-4664-b502-8ab797838fb8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jph2d"
Sep 17 23:10:20.771368 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:20.771269    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/123417a3-f884-4c2f-b7ac-837813412743-host-kubelet\") pod \"ovnkube-node-tcglq\" (UID: \"123417a3-f884-4c2f-b7ac-837813412743\") " pod="openshift-ovn-kubernetes/ovnkube-node-tcglq"
Sep 17 23:10:20.771368 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:20.771284    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/123417a3-f884-4c2f-b7ac-837813412743-ovnkube-config\") pod \"ovnkube-node-tcglq\" (UID: \"123417a3-f884-4c2f-b7ac-837813412743\") " pod="openshift-ovn-kubernetes/ovnkube-node-tcglq"
Sep 17 23:10:20.771368 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:20.771292    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/891302e3-91c1-4b56-a169-950bb8587aa5-tmp\") pod \"tuned-jf4g9\" (UID: \"891302e3-91c1-4b56-a169-950bb8587aa5\") " pod="openshift-cluster-node-tuning-operator/tuned-jf4g9"
Sep 17 23:10:20.771368 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:20.771298    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/123417a3-f884-4c2f-b7ac-837813412743-host-run-netns\") pod \"ovnkube-node-tcglq\" (UID: \"123417a3-f884-4c2f-b7ac-837813412743\") " pod="openshift-ovn-kubernetes/ovnkube-node-tcglq"
Sep 17 23:10:20.771662 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:20.771387    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/123417a3-f884-4c2f-b7ac-837813412743-var-lib-openvswitch\") pod \"ovnkube-node-tcglq\" (UID: \"123417a3-f884-4c2f-b7ac-837813412743\") " pod="openshift-ovn-kubernetes/ovnkube-node-tcglq"
Sep 17 23:10:20.771662 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:20.771418    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/123417a3-f884-4c2f-b7ac-837813412743-host-run-ovn-kubernetes\") pod \"ovnkube-node-tcglq\" (UID: \"123417a3-f884-4c2f-b7ac-837813412743\") " pod="openshift-ovn-kubernetes/ovnkube-node-tcglq"
Sep 17 23:10:20.771662 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:20.771447    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/123417a3-f884-4c2f-b7ac-837813412743-host-cni-netd\") pod \"ovnkube-node-tcglq\" (UID: \"123417a3-f884-4c2f-b7ac-837813412743\") " pod="openshift-ovn-kubernetes/ovnkube-node-tcglq"
Sep 17 23:10:20.771662 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:20.771477    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/123417a3-f884-4c2f-b7ac-837813412743-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-tcglq\" (UID: \"123417a3-f884-4c2f-b7ac-837813412743\") " pod="openshift-ovn-kubernetes/ovnkube-node-tcglq"
Sep 17 23:10:20.771662 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:20.771494    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/123417a3-f884-4c2f-b7ac-837813412743-run-ovn\") pod \"ovnkube-node-tcglq\" (UID: \"123417a3-f884-4c2f-b7ac-837813412743\") " pod="openshift-ovn-kubernetes/ovnkube-node-tcglq"
Sep 17 23:10:20.771662 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:20.771507    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/123417a3-f884-4c2f-b7ac-837813412743-ovnkube-script-lib\") pod \"ovnkube-node-tcglq\" (UID: \"123417a3-f884-4c2f-b7ac-837813412743\") " pod="openshift-ovn-kubernetes/ovnkube-node-tcglq"
Sep 17 23:10:20.771662 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:20.771532    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/891302e3-91c1-4b56-a169-950bb8587aa5-etc-modprobe-d\") pod \"tuned-jf4g9\" (UID: \"891302e3-91c1-4b56-a169-950bb8587aa5\") " pod="openshift-cluster-node-tuning-operator/tuned-jf4g9"
Sep 17 23:10:20.771662 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:20.771561    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ktgmc\" (UniqueName: \"kubernetes.io/projected/fcc7ae4e-ace4-490f-a4e0-1c7c0ea22f2d-kube-api-access-ktgmc\") pod \"node-resolver-895t6\" (UID: \"fcc7ae4e-ace4-490f-a4e0-1c7c0ea22f2d\") " pod="openshift-dns/node-resolver-895t6"
Sep 17 23:10:20.771662 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:20.771563    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/123417a3-f884-4c2f-b7ac-837813412743-host-kubelet\") pod \"ovnkube-node-tcglq\" (UID: \"123417a3-f884-4c2f-b7ac-837813412743\") " pod="openshift-ovn-kubernetes/ovnkube-node-tcglq"
Sep 17 23:10:20.771662 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:20.771590    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/04f96656-0988-4bea-a70f-79d1b8445201-serviceca\") pod \"node-ca-5spz7\" (UID: \"04f96656-0988-4bea-a70f-79d1b8445201\") " pod="openshift-image-registry/node-ca-5spz7"
Sep 17 23:10:20.771662 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:20.771618    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d672140f-ad59-4a68-9dde-80b62492af7f-metrics-certs\") pod \"network-metrics-daemon-4bbdp\" (UID: \"d672140f-ad59-4a68-9dde-80b62492af7f\") " pod="openshift-multus/network-metrics-daemon-4bbdp"
Sep 17 23:10:20.771662 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:20.771646    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l6p4l\" (UniqueName: \"kubernetes.io/projected/d672140f-ad59-4a68-9dde-80b62492af7f-kube-api-access-l6p4l\") pod \"network-metrics-daemon-4bbdp\" (UID: \"d672140f-ad59-4a68-9dde-80b62492af7f\") " pod="openshift-multus/network-metrics-daemon-4bbdp"
Sep 17 23:10:20.771662 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:20.771664    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/891302e3-91c1-4b56-a169-950bb8587aa5-etc-sysctl-d\") pod \"tuned-jf4g9\" (UID: \"891302e3-91c1-4b56-a169-950bb8587aa5\") " pod="openshift-cluster-node-tuning-operator/tuned-jf4g9"
Sep 17 23:10:20.772398 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:20.771672    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/123417a3-f884-4c2f-b7ac-837813412743-run-openvswitch\") pod \"ovnkube-node-tcglq\" (UID: \"123417a3-f884-4c2f-b7ac-837813412743\") " pod="openshift-ovn-kubernetes/ovnkube-node-tcglq"
Sep 17 23:10:20.772398 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:20.771706    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/123417a3-f884-4c2f-b7ac-837813412743-env-overrides\") pod \"ovnkube-node-tcglq\" (UID: \"123417a3-f884-4c2f-b7ac-837813412743\") " pod="openshift-ovn-kubernetes/ovnkube-node-tcglq"
Sep 17 23:10:20.772398 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:20.771712    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/1ae5e3b8-3cf3-4664-b502-8ab797838fb8-sys-fs\") pod \"aws-ebs-csi-driver-node-jph2d\" (UID: \"1ae5e3b8-3cf3-4664-b502-8ab797838fb8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jph2d"
Sep 17 23:10:20.772398 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:20.771733    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/891302e3-91c1-4b56-a169-950bb8587aa5-etc-tuned\") pod \"tuned-jf4g9\" (UID: \"891302e3-91c1-4b56-a169-950bb8587aa5\") " pod="openshift-cluster-node-tuning-operator/tuned-jf4g9"
Sep 17 23:10:20.772398 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:20.771762    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f6e69d33-8d35-4bbd-b113-e16f0b87f42c-host-slash\") pod \"iptables-alerter-4bls7\" (UID: \"f6e69d33-8d35-4bbd-b113-e16f0b87f42c\") " pod="openshift-network-operator/iptables-alerter-4bls7"
Sep 17 23:10:20.772398 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:20.771776    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/1c71bd0e-1761-4157-82d0-a33701aece84-multus-daemon-config\") pod \"multus-9j2wk\" (UID: \"1c71bd0e-1761-4157-82d0-a33701aece84\") " pod="openshift-multus/multus-9j2wk"
Sep 17 23:10:20.772398 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:20.771842    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/123417a3-f884-4c2f-b7ac-837813412743-host-run-netns\") pod \"ovnkube-node-tcglq\" (UID: \"123417a3-f884-4c2f-b7ac-837813412743\") " pod="openshift-ovn-kubernetes/ovnkube-node-tcglq"
Sep 17 23:10:20.772398 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:20.771949    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f6e69d33-8d35-4bbd-b113-e16f0b87f42c-host-slash\") pod \"iptables-alerter-4bls7\" (UID: \"f6e69d33-8d35-4bbd-b113-e16f0b87f42c\") " pod="openshift-network-operator/iptables-alerter-4bls7"
Sep 17 23:10:20.772398 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:20.771961    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/05765151-c799-4fe2-9f2e-cf4181e84ffa-konnectivity-ca\") pod \"konnectivity-agent-g2vzq\" (UID: \"05765151-c799-4fe2-9f2e-cf4181e84ffa\") " pod="kube-system/konnectivity-agent-g2vzq"
Sep 17 23:10:20.772398 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:20.771983    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/891302e3-91c1-4b56-a169-950bb8587aa5-etc-modprobe-d\") pod \"tuned-jf4g9\" (UID: \"891302e3-91c1-4b56-a169-950bb8587aa5\") " pod="openshift-cluster-node-tuning-operator/tuned-jf4g9"
Sep 17 23:10:20.772398 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:20.771997    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1c71bd0e-1761-4157-82d0-a33701aece84-cnibin\") pod \"multus-9j2wk\" (UID: \"1c71bd0e-1761-4157-82d0-a33701aece84\") " pod="openshift-multus/multus-9j2wk"
Sep 17 23:10:20.772398 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:20.772123    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/891302e3-91c1-4b56-a169-950bb8587aa5-etc-sysctl-conf\") pod \"tuned-jf4g9\" (UID: \"891302e3-91c1-4b56-a169-950bb8587aa5\") " pod="openshift-cluster-node-tuning-operator/tuned-jf4g9"
Sep 17 23:10:20.772398 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:20.772244    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ea8e5a0a-14f1-4be6-84e1-33567e5d6032-tuning-conf-dir\") pod \"multus-additional-cni-plugins-2gjg8\" (UID: \"ea8e5a0a-14f1-4be6-84e1-33567e5d6032\") " pod="openshift-multus/multus-additional-cni-plugins-2gjg8"
Sep 17 23:10:20.772398 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:20.772287    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/123417a3-f884-4c2f-b7ac-837813412743-var-lib-openvswitch\") pod \"ovnkube-node-tcglq\" (UID: \"123417a3-f884-4c2f-b7ac-837813412743\") " pod="openshift-ovn-kubernetes/ovnkube-node-tcglq"
Sep 17 23:10:20.772398 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:20.772324    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/123417a3-f884-4c2f-b7ac-837813412743-host-run-ovn-kubernetes\") pod \"ovnkube-node-tcglq\" (UID: \"123417a3-f884-4c2f-b7ac-837813412743\") " pod="openshift-ovn-kubernetes/ovnkube-node-tcglq"
Sep 17 23:10:20.772398 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:20.772362    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/123417a3-f884-4c2f-b7ac-837813412743-host-cni-netd\") pod \"ovnkube-node-tcglq\" (UID: \"123417a3-f884-4c2f-b7ac-837813412743\") " pod="openshift-ovn-kubernetes/ovnkube-node-tcglq"
Sep 17 23:10:20.772398 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:20.772366    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/123417a3-f884-4c2f-b7ac-837813412743-env-overrides\") pod \"ovnkube-node-tcglq\" (UID: \"123417a3-f884-4c2f-b7ac-837813412743\") " pod="openshift-ovn-kubernetes/ovnkube-node-tcglq"
Sep 17 23:10:20.773250 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:20.772399    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/123417a3-f884-4c2f-b7ac-837813412743-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-tcglq\" (UID: \"123417a3-f884-4c2f-b7ac-837813412743\") " pod="openshift-ovn-kubernetes/ovnkube-node-tcglq"
Sep 17 23:10:20.773250 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:20.772442    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/fcc7ae4e-ace4-490f-a4e0-1c7c0ea22f2d-hosts-file\") pod \"node-resolver-895t6\" (UID: \"fcc7ae4e-ace4-490f-a4e0-1c7c0ea22f2d\") " pod="openshift-dns/node-resolver-895t6"
Sep 17 23:10:20.773250 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:20.772468    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fcc7ae4e-ace4-490f-a4e0-1c7c0ea22f2d-tmp-dir\") pod \"node-resolver-895t6\" (UID: \"fcc7ae4e-ace4-490f-a4e0-1c7c0ea22f2d\") " pod="openshift-dns/node-resolver-895t6"
Sep 17 23:10:20.773250 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:20.772471    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/fa08be4f-e950-4dee-a227-feb39c2ba7e0-dbus\") pod \"global-pull-secret-syncer-24tsm\" (UID: \"fa08be4f-e950-4dee-a227-feb39c2ba7e0\") " pod="kube-system/global-pull-secret-syncer-24tsm"
Sep 17 23:10:20.773250 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:20.772498    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/1c71bd0e-1761-4157-82d0-a33701aece84-host-run-k8s-cni-cncf-io\") pod \"multus-9j2wk\" (UID: \"1c71bd0e-1761-4157-82d0-a33701aece84\") " pod="openshift-multus/multus-9j2wk"
Sep 17 23:10:20.773250 ip-10-0-142-4 kubenswrapper[2584]: E0917 23:10:20.772538    2584 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 23:10:20.773250 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:20.772589    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/123417a3-f884-4c2f-b7ac-837813412743-run-openvswitch\") pod \"ovnkube-node-tcglq\" (UID: \"123417a3-f884-4c2f-b7ac-837813412743\") " pod="openshift-ovn-kubernetes/ovnkube-node-tcglq"
Sep 17 23:10:20.773250 ip-10-0-142-4 kubenswrapper[2584]: E0917 23:10:20.772657    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d672140f-ad59-4a68-9dde-80b62492af7f-metrics-certs podName:d672140f-ad59-4a68-9dde-80b62492af7f nodeName:}" failed. No retries permitted until 2026-09-17 23:10:21.272641598 +0000 UTC m=+4.638301494 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d672140f-ad59-4a68-9dde-80b62492af7f-metrics-certs") pod "network-metrics-daemon-4bbdp" (UID: "d672140f-ad59-4a68-9dde-80b62492af7f") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 23:10:20.773250 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:20.772706    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/fcc7ae4e-ace4-490f-a4e0-1c7c0ea22f2d-hosts-file\") pod \"node-resolver-895t6\" (UID: \"fcc7ae4e-ace4-490f-a4e0-1c7c0ea22f2d\") " pod="openshift-dns/node-resolver-895t6"
Sep 17 23:10:20.773250 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:20.772743    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/123417a3-f884-4c2f-b7ac-837813412743-etc-openvswitch\") pod \"ovnkube-node-tcglq\" (UID: \"123417a3-f884-4c2f-b7ac-837813412743\") " pod="openshift-ovn-kubernetes/ovnkube-node-tcglq"
Sep 17 23:10:20.773250 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:20.772773    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/123417a3-f884-4c2f-b7ac-837813412743-ovn-node-metrics-cert\") pod \"ovnkube-node-tcglq\" (UID: \"123417a3-f884-4c2f-b7ac-837813412743\") " pod="openshift-ovn-kubernetes/ovnkube-node-tcglq"
Sep 17 23:10:20.773250 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:20.772594    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/1c71bd0e-1761-4157-82d0-a33701aece84-hostroot\") pod \"multus-9j2wk\" (UID: \"1c71bd0e-1761-4157-82d0-a33701aece84\") " pod="openshift-multus/multus-9j2wk"
Sep 17 23:10:20.773250 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:20.772871    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/123417a3-f884-4c2f-b7ac-837813412743-ovnkube-script-lib\") pod \"ovnkube-node-tcglq\" (UID: \"123417a3-f884-4c2f-b7ac-837813412743\") " pod="openshift-ovn-kubernetes/ovnkube-node-tcglq"
Sep 17 23:10:20.773250 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:20.772982    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1c71bd0e-1761-4157-82d0-a33701aece84-etc-kubernetes\") pod \"multus-9j2wk\" (UID: \"1c71bd0e-1761-4157-82d0-a33701aece84\") " pod="openshift-multus/multus-9j2wk"
Sep 17 23:10:20.773250 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:20.773042    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fcc7ae4e-ace4-490f-a4e0-1c7c0ea22f2d-tmp-dir\") pod \"node-resolver-895t6\" (UID: \"fcc7ae4e-ace4-490f-a4e0-1c7c0ea22f2d\") " pod="openshift-dns/node-resolver-895t6"
Sep 17 23:10:20.773250 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:20.773104    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/891302e3-91c1-4b56-a169-950bb8587aa5-lib-modules\") pod \"tuned-jf4g9\" (UID: \"891302e3-91c1-4b56-a169-950bb8587aa5\") " pod="openshift-cluster-node-tuning-operator/tuned-jf4g9"
Sep 17 23:10:20.773250 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:20.773117    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/1c71bd0e-1761-4157-82d0-a33701aece84-host-run-k8s-cni-cncf-io\") pod \"multus-9j2wk\" (UID: \"1c71bd0e-1761-4157-82d0-a33701aece84\") " pod="openshift-multus/multus-9j2wk"
Sep 17 23:10:20.774142 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:20.773137    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f6e69d33-8d35-4bbd-b113-e16f0b87f42c-iptables-alerter-script\") pod \"iptables-alerter-4bls7\" (UID: \"f6e69d33-8d35-4bbd-b113-e16f0b87f42c\") " pod="openshift-network-operator/iptables-alerter-4bls7"
Sep 17 23:10:20.774142 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:20.773171    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/04f96656-0988-4bea-a70f-79d1b8445201-serviceca\") pod \"node-ca-5spz7\" (UID: \"04f96656-0988-4bea-a70f-79d1b8445201\") " pod="openshift-image-registry/node-ca-5spz7"
Sep 17 23:10:20.774142 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:20.773330    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1c71bd0e-1761-4157-82d0-a33701aece84-cni-binary-copy\") pod \"multus-9j2wk\" (UID: \"1c71bd0e-1761-4157-82d0-a33701aece84\") " pod="openshift-multus/multus-9j2wk"
Sep 17 23:10:20.774142 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:20.773400    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1c71bd0e-1761-4157-82d0-a33701aece84-multus-conf-dir\") pod \"multus-9j2wk\" (UID: \"1c71bd0e-1761-4157-82d0-a33701aece84\") " pod="openshift-multus/multus-9j2wk"
Sep 17 23:10:20.774142 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:20.773432    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1c71bd0e-1761-4157-82d0-a33701aece84-multus-cni-dir\") pod \"multus-9j2wk\" (UID: \"1c71bd0e-1761-4157-82d0-a33701aece84\") " pod="openshift-multus/multus-9j2wk"
Sep 17 23:10:20.774142 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:20.773457    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1c71bd0e-1761-4157-82d0-a33701aece84-host-run-netns\") pod \"multus-9j2wk\" (UID: \"1c71bd0e-1761-4157-82d0-a33701aece84\") " pod="openshift-multus/multus-9j2wk"
Sep 17 23:10:20.774142 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:20.773485    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/04f96656-0988-4bea-a70f-79d1b8445201-host\") pod \"node-ca-5spz7\" (UID: \"04f96656-0988-4bea-a70f-79d1b8445201\") " pod="openshift-image-registry/node-ca-5spz7"
Sep 17 23:10:20.774142 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:20.773551    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/05765151-c799-4fe2-9f2e-cf4181e84ffa-agent-certs\") pod \"konnectivity-agent-g2vzq\" (UID: \"05765151-c799-4fe2-9f2e-cf4181e84ffa\") " pod="kube-system/konnectivity-agent-g2vzq"
Sep 17 23:10:20.774142 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:20.774037    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/ea8e5a0a-14f1-4be6-84e1-33567e5d6032-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-2gjg8\" (UID: \"ea8e5a0a-14f1-4be6-84e1-33567e5d6032\") " pod="openshift-multus/multus-additional-cni-plugins-2gjg8"
Sep 17 23:10:20.774142 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:20.774131    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1c71bd0e-1761-4157-82d0-a33701aece84-multus-cni-dir\") pod \"multus-9j2wk\" (UID: \"1c71bd0e-1761-4157-82d0-a33701aece84\") " pod="openshift-multus/multus-9j2wk"
Sep 17 23:10:20.774610 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:20.774183    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1c71bd0e-1761-4157-82d0-a33701aece84-multus-conf-dir\") pod \"multus-9j2wk\" (UID: \"1c71bd0e-1761-4157-82d0-a33701aece84\") " pod="openshift-multus/multus-9j2wk"
Sep 17 23:10:20.774610 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:20.774235    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1c71bd0e-1761-4157-82d0-a33701aece84-host-run-netns\") pod \"multus-9j2wk\" (UID: \"1c71bd0e-1761-4157-82d0-a33701aece84\") " pod="openshift-multus/multus-9j2wk"
Sep 17 23:10:20.774610 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:20.774291    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/04f96656-0988-4bea-a70f-79d1b8445201-host\") pod \"node-ca-5spz7\" (UID: \"04f96656-0988-4bea-a70f-79d1b8445201\") " pod="openshift-image-registry/node-ca-5spz7"
Sep 17 23:10:20.774610 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:20.774343    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/123417a3-f884-4c2f-b7ac-837813412743-etc-openvswitch\") pod \"ovnkube-node-tcglq\" (UID: \"123417a3-f884-4c2f-b7ac-837813412743\") " pod="openshift-ovn-kubernetes/ovnkube-node-tcglq"
Sep 17 23:10:20.774610 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:20.774382    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f6e69d33-8d35-4bbd-b113-e16f0b87f42c-iptables-alerter-script\") pod \"iptables-alerter-4bls7\" (UID: \"f6e69d33-8d35-4bbd-b113-e16f0b87f42c\") " pod="openshift-network-operator/iptables-alerter-4bls7"
Sep 17 23:10:20.774610 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:20.774513    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/891302e3-91c1-4b56-a169-950bb8587aa5-lib-modules\") pod \"tuned-jf4g9\" (UID: \"891302e3-91c1-4b56-a169-950bb8587aa5\") " pod="openshift-cluster-node-tuning-operator/tuned-jf4g9"
Sep 17 23:10:20.775783 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:20.775724    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/123417a3-f884-4c2f-b7ac-837813412743-ovn-node-metrics-cert\") pod \"ovnkube-node-tcglq\" (UID: \"123417a3-f884-4c2f-b7ac-837813412743\") " pod="openshift-ovn-kubernetes/ovnkube-node-tcglq"
Sep 17 23:10:20.775931 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:20.775913    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/891302e3-91c1-4b56-a169-950bb8587aa5-etc-tuned\") pod \"tuned-jf4g9\" (UID: \"891302e3-91c1-4b56-a169-950bb8587aa5\") " pod="openshift-cluster-node-tuning-operator/tuned-jf4g9"
Sep 17 23:10:20.777699 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:20.777678    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/05765151-c799-4fe2-9f2e-cf4181e84ffa-agent-certs\") pod \"konnectivity-agent-g2vzq\" (UID: \"05765151-c799-4fe2-9f2e-cf4181e84ffa\") " pod="kube-system/konnectivity-agent-g2vzq"
Sep 17 23:10:20.783061 ip-10-0-142-4 kubenswrapper[2584]: E0917 23:10:20.782313    2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 23:10:20.783061 ip-10-0-142-4 kubenswrapper[2584]: E0917 23:10:20.782339    2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 23:10:20.783061 ip-10-0-142-4 kubenswrapper[2584]: E0917 23:10:20.782351    2584 projected.go:196] Error preparing data for projected volume kube-api-access-bgsh5 for pod openshift-network-diagnostics/network-check-target-wgps9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 23:10:20.783061 ip-10-0-142-4 kubenswrapper[2584]: E0917 23:10:20.782406    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a2a26114-4c24-4dce-929c-3cb5caf17e6a-kube-api-access-bgsh5 podName:a2a26114-4c24-4dce-929c-3cb5caf17e6a nodeName:}" failed. No retries permitted until 2026-09-17 23:10:21.282389687 +0000 UTC m=+4.648049582 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-bgsh5" (UniqueName: "kubernetes.io/projected/a2a26114-4c24-4dce-929c-3cb5caf17e6a-kube-api-access-bgsh5") pod "network-check-target-wgps9" (UID: "a2a26114-4c24-4dce-929c-3cb5caf17e6a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 23:10:20.788404 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:20.788382    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7xl4v\" (UniqueName: \"kubernetes.io/projected/891302e3-91c1-4b56-a169-950bb8587aa5-kube-api-access-7xl4v\") pod \"tuned-jf4g9\" (UID: \"891302e3-91c1-4b56-a169-950bb8587aa5\") " pod="openshift-cluster-node-tuning-operator/tuned-jf4g9"
Sep 17 23:10:20.791520 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:20.791498    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ktgmc\" (UniqueName: \"kubernetes.io/projected/fcc7ae4e-ace4-490f-a4e0-1c7c0ea22f2d-kube-api-access-ktgmc\") pod \"node-resolver-895t6\" (UID: \"fcc7ae4e-ace4-490f-a4e0-1c7c0ea22f2d\") " pod="openshift-dns/node-resolver-895t6"
Sep 17 23:10:20.792404 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:20.792373    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x7h7k\" (UniqueName: \"kubernetes.io/projected/123417a3-f884-4c2f-b7ac-837813412743-kube-api-access-x7h7k\") pod \"ovnkube-node-tcglq\" (UID: \"123417a3-f884-4c2f-b7ac-837813412743\") " pod="openshift-ovn-kubernetes/ovnkube-node-tcglq"
Sep 17 23:10:20.793096 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:20.793074    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zfbcm\" (UniqueName: \"kubernetes.io/projected/f6e69d33-8d35-4bbd-b113-e16f0b87f42c-kube-api-access-zfbcm\") pod \"iptables-alerter-4bls7\" (UID: \"f6e69d33-8d35-4bbd-b113-e16f0b87f42c\") " pod="openshift-network-operator/iptables-alerter-4bls7"
Sep 17 23:10:20.793418 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:20.793399    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l6p4l\" (UniqueName: \"kubernetes.io/projected/d672140f-ad59-4a68-9dde-80b62492af7f-kube-api-access-l6p4l\") pod \"network-metrics-daemon-4bbdp\" (UID: \"d672140f-ad59-4a68-9dde-80b62492af7f\") " pod="openshift-multus/network-metrics-daemon-4bbdp"
Sep 17 23:10:20.793930 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:20.793909    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cwfm4\" (UniqueName: \"kubernetes.io/projected/04f96656-0988-4bea-a70f-79d1b8445201-kube-api-access-cwfm4\") pod \"node-ca-5spz7\" (UID: \"04f96656-0988-4bea-a70f-79d1b8445201\") " pod="openshift-image-registry/node-ca-5spz7"
Sep 17 23:10:20.794262 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:20.794246    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zmhgw\" (UniqueName: \"kubernetes.io/projected/1c71bd0e-1761-4157-82d0-a33701aece84-kube-api-access-zmhgw\") pod \"multus-9j2wk\" (UID: \"1c71bd0e-1761-4157-82d0-a33701aece84\") " pod="openshift-multus/multus-9j2wk"
Sep 17 23:10:20.794946 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:20.794925    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wntvp\" (UniqueName: \"kubernetes.io/projected/1ae5e3b8-3cf3-4664-b502-8ab797838fb8-kube-api-access-wntvp\") pod \"aws-ebs-csi-driver-node-jph2d\" (UID: \"1ae5e3b8-3cf3-4664-b502-8ab797838fb8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jph2d"
Sep 17 23:10:20.795054 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:20.795038    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5lhqm\" (UniqueName: \"kubernetes.io/projected/ea8e5a0a-14f1-4be6-84e1-33567e5d6032-kube-api-access-5lhqm\") pod \"multus-additional-cni-plugins-2gjg8\" (UID: \"ea8e5a0a-14f1-4be6-84e1-33567e5d6032\") " pod="openshift-multus/multus-additional-cni-plugins-2gjg8"
Sep 17 23:10:20.927242 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:20.927212    2584 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 23:10:20.934228 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:20.934194    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-tcglq"
Sep 17 23:10:20.941650 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:20.941628    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-9j2wk"
Sep 17 23:10:20.951690 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:20.951673    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-jf4g9"
Sep 17 23:10:20.958236 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:20.958219    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-4bls7"
Sep 17 23:10:20.968815 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:20.968799    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jph2d"
Sep 17 23:10:20.975465 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:20.975450    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-895t6"
Sep 17 23:10:20.982929 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:20.982910    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-5spz7"
Sep 17 23:10:20.991284 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:20.991265    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-g2vzq"
Sep 17 23:10:20.995863 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:20.995849    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-2gjg8"
Sep 17 23:10:21.276771 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:21.276738    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fa08be4f-e950-4dee-a227-feb39c2ba7e0-original-pull-secret\") pod \"global-pull-secret-syncer-24tsm\" (UID: \"fa08be4f-e950-4dee-a227-feb39c2ba7e0\") " pod="kube-system/global-pull-secret-syncer-24tsm"
Sep 17 23:10:21.276942 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:21.276784    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d672140f-ad59-4a68-9dde-80b62492af7f-metrics-certs\") pod \"network-metrics-daemon-4bbdp\" (UID: \"d672140f-ad59-4a68-9dde-80b62492af7f\") " pod="openshift-multus/network-metrics-daemon-4bbdp"
Sep 17 23:10:21.276942 ip-10-0-142-4 kubenswrapper[2584]: E0917 23:10:21.276887    2584 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 23:10:21.276942 ip-10-0-142-4 kubenswrapper[2584]: E0917 23:10:21.276887    2584 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 23:10:21.277105 ip-10-0-142-4 kubenswrapper[2584]: E0917 23:10:21.276954    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d672140f-ad59-4a68-9dde-80b62492af7f-metrics-certs podName:d672140f-ad59-4a68-9dde-80b62492af7f nodeName:}" failed. No retries permitted until 2026-09-17 23:10:22.276936642 +0000 UTC m=+5.642596550 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d672140f-ad59-4a68-9dde-80b62492af7f-metrics-certs") pod "network-metrics-daemon-4bbdp" (UID: "d672140f-ad59-4a68-9dde-80b62492af7f") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 23:10:21.277105 ip-10-0-142-4 kubenswrapper[2584]: E0917 23:10:21.276971    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fa08be4f-e950-4dee-a227-feb39c2ba7e0-original-pull-secret podName:fa08be4f-e950-4dee-a227-feb39c2ba7e0 nodeName:}" failed. No retries permitted until 2026-09-17 23:10:22.276962457 +0000 UTC m=+5.642622365 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fa08be4f-e950-4dee-a227-feb39c2ba7e0-original-pull-secret") pod "global-pull-secret-syncer-24tsm" (UID: "fa08be4f-e950-4dee-a227-feb39c2ba7e0") : object "kube-system"/"original-pull-secret" not registered
Sep 17 23:10:21.377534 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:21.377509    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bgsh5\" (UniqueName: \"kubernetes.io/projected/a2a26114-4c24-4dce-929c-3cb5caf17e6a-kube-api-access-bgsh5\") pod \"network-check-target-wgps9\" (UID: \"a2a26114-4c24-4dce-929c-3cb5caf17e6a\") " pod="openshift-network-diagnostics/network-check-target-wgps9"
Sep 17 23:10:21.377686 ip-10-0-142-4 kubenswrapper[2584]: E0917 23:10:21.377664    2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 23:10:21.377686 ip-10-0-142-4 kubenswrapper[2584]: E0917 23:10:21.377686    2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 23:10:21.377765 ip-10-0-142-4 kubenswrapper[2584]: E0917 23:10:21.377698    2584 projected.go:196] Error preparing data for projected volume kube-api-access-bgsh5 for pod openshift-network-diagnostics/network-check-target-wgps9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 23:10:21.377765 ip-10-0-142-4 kubenswrapper[2584]: E0917 23:10:21.377754    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a2a26114-4c24-4dce-929c-3cb5caf17e6a-kube-api-access-bgsh5 podName:a2a26114-4c24-4dce-929c-3cb5caf17e6a nodeName:}" failed. No retries permitted until 2026-09-17 23:10:22.377738771 +0000 UTC m=+5.743398670 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-bgsh5" (UniqueName: "kubernetes.io/projected/a2a26114-4c24-4dce-929c-3cb5caf17e6a-kube-api-access-bgsh5") pod "network-check-target-wgps9" (UID: "a2a26114-4c24-4dce-929c-3cb5caf17e6a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 23:10:21.587145 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:21.587072    2584 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 23:10:21.726407 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:21.726371    2584 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-16 23:05:20 +0000 UTC" deadline="2028-05-21 05:29:15.493644793 +0000 UTC"
Sep 17 23:10:21.726407 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:21.726405    2584 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14670h18m53.767244939s"
Sep 17 23:10:21.802160 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:21.802136    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfcc7ae4e_ace4_490f_a4e0_1c7c0ea22f2d.slice/crio-0f6ccdbab5c6814eb0dc407583870586a46c3f34ac4830379a498bb5289bf53e WatchSource:0}: Error finding container 0f6ccdbab5c6814eb0dc407583870586a46c3f34ac4830379a498bb5289bf53e: Status 404 returned error can't find the container with id 0f6ccdbab5c6814eb0dc407583870586a46c3f34ac4830379a498bb5289bf53e
Sep 17 23:10:21.802856 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:21.802833    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6e69d33_8d35_4bbd_b113_e16f0b87f42c.slice/crio-9853f57e164e19c02ce8780b12a1053eb66bded7b0c52605a8402be7e98df57b WatchSource:0}: Error finding container 9853f57e164e19c02ce8780b12a1053eb66bded7b0c52605a8402be7e98df57b: Status 404 returned error can't find the container with id 9853f57e164e19c02ce8780b12a1053eb66bded7b0c52605a8402be7e98df57b
Sep 17 23:10:21.803500 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:21.803478    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea8e5a0a_14f1_4be6_84e1_33567e5d6032.slice/crio-34c8ef23e9dce8f1ddf4b90cff29f8b47cdca5951b2323441af3eecbc3e04de4 WatchSource:0}: Error finding container 34c8ef23e9dce8f1ddf4b90cff29f8b47cdca5951b2323441af3eecbc3e04de4: Status 404 returned error can't find the container with id 34c8ef23e9dce8f1ddf4b90cff29f8b47cdca5951b2323441af3eecbc3e04de4
Sep 17 23:10:21.804923 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:21.804335    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05765151_c799_4fe2_9f2e_cf4181e84ffa.slice/crio-391fee1b285473b9fdfdac70c707bec596c11c22b9f0a36f347b8531f93c2a68 WatchSource:0}: Error finding container 391fee1b285473b9fdfdac70c707bec596c11c22b9f0a36f347b8531f93c2a68: Status 404 returned error can't find the container with id 391fee1b285473b9fdfdac70c707bec596c11c22b9f0a36f347b8531f93c2a68
Sep 17 23:10:21.809424 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:21.809403    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod123417a3_f884_4c2f_b7ac_837813412743.slice/crio-de7d0b9bc8dd215c6b467a4a465f23c0ba2e21fafec4c5b7db09a6938a18d7c3 WatchSource:0}: Error finding container de7d0b9bc8dd215c6b467a4a465f23c0ba2e21fafec4c5b7db09a6938a18d7c3: Status 404 returned error can't find the container with id de7d0b9bc8dd215c6b467a4a465f23c0ba2e21fafec4c5b7db09a6938a18d7c3
Sep 17 23:10:21.809962 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:21.809802    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04f96656_0988_4bea_a70f_79d1b8445201.slice/crio-5573aff52bb963940abc8ef01022625c38494c2b6a9c0612e4649d7996faa2a1 WatchSource:0}: Error finding container 5573aff52bb963940abc8ef01022625c38494c2b6a9c0612e4649d7996faa2a1: Status 404 returned error can't find the container with id 5573aff52bb963940abc8ef01022625c38494c2b6a9c0612e4649d7996faa2a1
Sep 17 23:10:21.812120 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:21.811828    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c71bd0e_1761_4157_82d0_a33701aece84.slice/crio-ffbaea85239cd07cda78390acd08d8b6a4925fa057e7c73bd515eee3b0ac77b7 WatchSource:0}: Error finding container ffbaea85239cd07cda78390acd08d8b6a4925fa057e7c73bd515eee3b0ac77b7: Status 404 returned error can't find the container with id ffbaea85239cd07cda78390acd08d8b6a4925fa057e7c73bd515eee3b0ac77b7
Sep 17 23:10:21.813540 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:21.813073    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod891302e3_91c1_4b56_a169_950bb8587aa5.slice/crio-e527e7ddbab8413f66963c7bcc5d63d912830a39ab3a901918333308c6b3a739 WatchSource:0}: Error finding container e527e7ddbab8413f66963c7bcc5d63d912830a39ab3a901918333308c6b3a739: Status 404 returned error can't find the container with id e527e7ddbab8413f66963c7bcc5d63d912830a39ab3a901918333308c6b3a739
Sep 17 23:10:21.814608 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:21.814584    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ae5e3b8_3cf3_4664_b502_8ab797838fb8.slice/crio-e3bc29d88dcff7ed4305a5dc54010200fd3994cd5dbc6ce76702b2d46c8e7981 WatchSource:0}: Error finding container e3bc29d88dcff7ed4305a5dc54010200fd3994cd5dbc6ce76702b2d46c8e7981: Status 404 returned error can't find the container with id e3bc29d88dcff7ed4305a5dc54010200fd3994cd5dbc6ce76702b2d46c8e7981
Sep 17 23:10:22.282237 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:22.282201    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fa08be4f-e950-4dee-a227-feb39c2ba7e0-original-pull-secret\") pod \"global-pull-secret-syncer-24tsm\" (UID: \"fa08be4f-e950-4dee-a227-feb39c2ba7e0\") " pod="kube-system/global-pull-secret-syncer-24tsm"
Sep 17 23:10:22.282417 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:22.282269    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d672140f-ad59-4a68-9dde-80b62492af7f-metrics-certs\") pod \"network-metrics-daemon-4bbdp\" (UID: \"d672140f-ad59-4a68-9dde-80b62492af7f\") " pod="openshift-multus/network-metrics-daemon-4bbdp"
Sep 17 23:10:22.282417 ip-10-0-142-4 kubenswrapper[2584]: E0917 23:10:22.282410    2584 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 23:10:22.282530 ip-10-0-142-4 kubenswrapper[2584]: E0917 23:10:22.282465    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d672140f-ad59-4a68-9dde-80b62492af7f-metrics-certs podName:d672140f-ad59-4a68-9dde-80b62492af7f nodeName:}" failed. No retries permitted until 2026-09-17 23:10:24.282447178 +0000 UTC m=+7.648107079 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d672140f-ad59-4a68-9dde-80b62492af7f-metrics-certs") pod "network-metrics-daemon-4bbdp" (UID: "d672140f-ad59-4a68-9dde-80b62492af7f") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 23:10:22.282724 ip-10-0-142-4 kubenswrapper[2584]: E0917 23:10:22.282697    2584 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 23:10:22.282814 ip-10-0-142-4 kubenswrapper[2584]: E0917 23:10:22.282764    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fa08be4f-e950-4dee-a227-feb39c2ba7e0-original-pull-secret podName:fa08be4f-e950-4dee-a227-feb39c2ba7e0 nodeName:}" failed. No retries permitted until 2026-09-17 23:10:24.282746587 +0000 UTC m=+7.648406493 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fa08be4f-e950-4dee-a227-feb39c2ba7e0-original-pull-secret") pod "global-pull-secret-syncer-24tsm" (UID: "fa08be4f-e950-4dee-a227-feb39c2ba7e0") : object "kube-system"/"original-pull-secret" not registered
Sep 17 23:10:22.383298 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:22.383265    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bgsh5\" (UniqueName: \"kubernetes.io/projected/a2a26114-4c24-4dce-929c-3cb5caf17e6a-kube-api-access-bgsh5\") pod \"network-check-target-wgps9\" (UID: \"a2a26114-4c24-4dce-929c-3cb5caf17e6a\") " pod="openshift-network-diagnostics/network-check-target-wgps9"
Sep 17 23:10:22.383498 ip-10-0-142-4 kubenswrapper[2584]: E0917 23:10:22.383473    2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 23:10:22.383498 ip-10-0-142-4 kubenswrapper[2584]: E0917 23:10:22.383495    2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 23:10:22.383633 ip-10-0-142-4 kubenswrapper[2584]: E0917 23:10:22.383509    2584 projected.go:196] Error preparing data for projected volume kube-api-access-bgsh5 for pod openshift-network-diagnostics/network-check-target-wgps9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 23:10:22.383633 ip-10-0-142-4 kubenswrapper[2584]: E0917 23:10:22.383564    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a2a26114-4c24-4dce-929c-3cb5caf17e6a-kube-api-access-bgsh5 podName:a2a26114-4c24-4dce-929c-3cb5caf17e6a nodeName:}" failed. No retries permitted until 2026-09-17 23:10:24.383546361 +0000 UTC m=+7.749206273 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-bgsh5" (UniqueName: "kubernetes.io/projected/a2a26114-4c24-4dce-929c-3cb5caf17e6a-kube-api-access-bgsh5") pod "network-check-target-wgps9" (UID: "a2a26114-4c24-4dce-929c-3cb5caf17e6a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 23:10:22.726803 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:22.726666    2584 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-16 23:05:20 +0000 UTC" deadline="2028-06-30 10:29:33.286676078 +0000 UTC"
Sep 17 23:10:22.726803 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:22.726722    2584 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15635h19m10.559958764s"
Sep 17 23:10:22.761044 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:22.760241    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-24tsm"
Sep 17 23:10:22.761044 ip-10-0-142-4 kubenswrapper[2584]: E0917 23:10:22.760366    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-24tsm" podUID="fa08be4f-e950-4dee-a227-feb39c2ba7e0"
Sep 17 23:10:22.761044 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:22.760755    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4bbdp"
Sep 17 23:10:22.761044 ip-10-0-142-4 kubenswrapper[2584]: E0917 23:10:22.760855    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4bbdp" podUID="d672140f-ad59-4a68-9dde-80b62492af7f"
Sep 17 23:10:22.761044 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:22.760913    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wgps9"
Sep 17 23:10:22.761044 ip-10-0-142-4 kubenswrapper[2584]: E0917 23:10:22.760978    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wgps9" podUID="a2a26114-4c24-4dce-929c-3cb5caf17e6a"
Sep 17 23:10:22.786525 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:22.786494    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jph2d" event={"ID":"1ae5e3b8-3cf3-4664-b502-8ab797838fb8","Type":"ContainerStarted","Data":"e3bc29d88dcff7ed4305a5dc54010200fd3994cd5dbc6ce76702b2d46c8e7981"}
Sep 17 23:10:22.808454 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:22.808423    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-5spz7" event={"ID":"04f96656-0988-4bea-a70f-79d1b8445201","Type":"ContainerStarted","Data":"5573aff52bb963940abc8ef01022625c38494c2b6a9c0612e4649d7996faa2a1"}
Sep 17 23:10:22.815577 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:22.815519    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-g2vzq" event={"ID":"05765151-c799-4fe2-9f2e-cf4181e84ffa","Type":"ContainerStarted","Data":"391fee1b285473b9fdfdac70c707bec596c11c22b9f0a36f347b8531f93c2a68"}
Sep 17 23:10:22.820593 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:22.820518    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-4bls7" event={"ID":"f6e69d33-8d35-4bbd-b113-e16f0b87f42c","Type":"ContainerStarted","Data":"9853f57e164e19c02ce8780b12a1053eb66bded7b0c52605a8402be7e98df57b"}
Sep 17 23:10:22.857872 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:22.854995    2584 generic.go:356] "Generic (PLEG): container finished" podID="68edadcfa29cc8be7e22dcaea45c88a8" containerID="f5a46e8d5be3835bfc520334f6e76823f2754d3a2aa8c8099c1bd7cb87b63808" exitCode=0
Sep 17 23:10:22.857872 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:22.855115    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-4.ec2.internal" event={"ID":"68edadcfa29cc8be7e22dcaea45c88a8","Type":"ContainerDied","Data":"f5a46e8d5be3835bfc520334f6e76823f2754d3a2aa8c8099c1bd7cb87b63808"}
Sep 17 23:10:22.863893 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:22.863866    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-4.ec2.internal" event={"ID":"3771e4c220b5a223d0f484f25cdc29e8","Type":"ContainerStarted","Data":"a964cbadaa84a38bebe612e73a77a0426f289c1befe3cd922f2a137de3fb8759"}
Sep 17 23:10:22.872541 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:22.872516    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-jf4g9" event={"ID":"891302e3-91c1-4b56-a169-950bb8587aa5","Type":"ContainerStarted","Data":"e527e7ddbab8413f66963c7bcc5d63d912830a39ab3a901918333308c6b3a739"}
Sep 17 23:10:22.888787 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:22.886751    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-9j2wk" event={"ID":"1c71bd0e-1761-4157-82d0-a33701aece84","Type":"ContainerStarted","Data":"ffbaea85239cd07cda78390acd08d8b6a4925fa057e7c73bd515eee3b0ac77b7"}
Sep 17 23:10:22.898956 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:22.898924    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tcglq" event={"ID":"123417a3-f884-4c2f-b7ac-837813412743","Type":"ContainerStarted","Data":"de7d0b9bc8dd215c6b467a4a465f23c0ba2e21fafec4c5b7db09a6938a18d7c3"}
Sep 17 23:10:22.903776 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:22.903725    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-4.ec2.internal" podStartSLOduration=3.903707691 podStartE2EDuration="3.903707691s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 23:10:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:10:22.902981017 +0000 UTC m=+6.268640930" watchObservedRunningTime="2026-09-17 23:10:22.903707691 +0000 UTC m=+6.269367626"
Sep 17 23:10:22.912553 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:22.912528    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2gjg8" event={"ID":"ea8e5a0a-14f1-4be6-84e1-33567e5d6032","Type":"ContainerStarted","Data":"34c8ef23e9dce8f1ddf4b90cff29f8b47cdca5951b2323441af3eecbc3e04de4"}
Sep 17 23:10:22.932698 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:22.932669    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-895t6" event={"ID":"fcc7ae4e-ace4-490f-a4e0-1c7c0ea22f2d","Type":"ContainerStarted","Data":"0f6ccdbab5c6814eb0dc407583870586a46c3f34ac4830379a498bb5289bf53e"}
Sep 17 23:10:23.964839 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:23.964591    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-4.ec2.internal" event={"ID":"68edadcfa29cc8be7e22dcaea45c88a8","Type":"ContainerStarted","Data":"edc4305fe2ada543d88a3712b1543dd556625bfb5a7209152db368623afc5796"}
Sep 17 23:10:24.303477 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:24.302705    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d672140f-ad59-4a68-9dde-80b62492af7f-metrics-certs\") pod \"network-metrics-daemon-4bbdp\" (UID: \"d672140f-ad59-4a68-9dde-80b62492af7f\") " pod="openshift-multus/network-metrics-daemon-4bbdp"
Sep 17 23:10:24.303477 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:24.302791    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fa08be4f-e950-4dee-a227-feb39c2ba7e0-original-pull-secret\") pod \"global-pull-secret-syncer-24tsm\" (UID: \"fa08be4f-e950-4dee-a227-feb39c2ba7e0\") " pod="kube-system/global-pull-secret-syncer-24tsm"
Sep 17 23:10:24.303477 ip-10-0-142-4 kubenswrapper[2584]: E0917 23:10:24.302909    2584 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 23:10:24.303477 ip-10-0-142-4 kubenswrapper[2584]: E0917 23:10:24.302982    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fa08be4f-e950-4dee-a227-feb39c2ba7e0-original-pull-secret podName:fa08be4f-e950-4dee-a227-feb39c2ba7e0 nodeName:}" failed. No retries permitted until 2026-09-17 23:10:28.302954148 +0000 UTC m=+11.668614066 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fa08be4f-e950-4dee-a227-feb39c2ba7e0-original-pull-secret") pod "global-pull-secret-syncer-24tsm" (UID: "fa08be4f-e950-4dee-a227-feb39c2ba7e0") : object "kube-system"/"original-pull-secret" not registered
Sep 17 23:10:24.303477 ip-10-0-142-4 kubenswrapper[2584]: E0917 23:10:24.303393    2584 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 23:10:24.303477 ip-10-0-142-4 kubenswrapper[2584]: E0917 23:10:24.303445    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d672140f-ad59-4a68-9dde-80b62492af7f-metrics-certs podName:d672140f-ad59-4a68-9dde-80b62492af7f nodeName:}" failed. No retries permitted until 2026-09-17 23:10:28.30342918 +0000 UTC m=+11.669089081 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d672140f-ad59-4a68-9dde-80b62492af7f-metrics-certs") pod "network-metrics-daemon-4bbdp" (UID: "d672140f-ad59-4a68-9dde-80b62492af7f") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 23:10:24.403858 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:24.403272    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bgsh5\" (UniqueName: \"kubernetes.io/projected/a2a26114-4c24-4dce-929c-3cb5caf17e6a-kube-api-access-bgsh5\") pod \"network-check-target-wgps9\" (UID: \"a2a26114-4c24-4dce-929c-3cb5caf17e6a\") " pod="openshift-network-diagnostics/network-check-target-wgps9"
Sep 17 23:10:24.403858 ip-10-0-142-4 kubenswrapper[2584]: E0917 23:10:24.403449    2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 23:10:24.403858 ip-10-0-142-4 kubenswrapper[2584]: E0917 23:10:24.403468    2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 23:10:24.403858 ip-10-0-142-4 kubenswrapper[2584]: E0917 23:10:24.403481    2584 projected.go:196] Error preparing data for projected volume kube-api-access-bgsh5 for pod openshift-network-diagnostics/network-check-target-wgps9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 23:10:24.403858 ip-10-0-142-4 kubenswrapper[2584]: E0917 23:10:24.403536    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a2a26114-4c24-4dce-929c-3cb5caf17e6a-kube-api-access-bgsh5 podName:a2a26114-4c24-4dce-929c-3cb5caf17e6a nodeName:}" failed. No retries permitted until 2026-09-17 23:10:28.403519394 +0000 UTC m=+11.769179292 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-bgsh5" (UniqueName: "kubernetes.io/projected/a2a26114-4c24-4dce-929c-3cb5caf17e6a-kube-api-access-bgsh5") pod "network-check-target-wgps9" (UID: "a2a26114-4c24-4dce-929c-3cb5caf17e6a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 23:10:24.762885 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:24.762057    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4bbdp"
Sep 17 23:10:24.762885 ip-10-0-142-4 kubenswrapper[2584]: E0917 23:10:24.762190    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4bbdp" podUID="d672140f-ad59-4a68-9dde-80b62492af7f"
Sep 17 23:10:24.762885 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:24.762547    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wgps9"
Sep 17 23:10:24.762885 ip-10-0-142-4 kubenswrapper[2584]: E0917 23:10:24.762630    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wgps9" podUID="a2a26114-4c24-4dce-929c-3cb5caf17e6a"
Sep 17 23:10:24.762885 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:24.762669    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-24tsm"
Sep 17 23:10:24.762885 ip-10-0-142-4 kubenswrapper[2584]: E0917 23:10:24.762732    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-24tsm" podUID="fa08be4f-e950-4dee-a227-feb39c2ba7e0"
Sep 17 23:10:26.759567 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:26.759126    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-24tsm"
Sep 17 23:10:26.759567 ip-10-0-142-4 kubenswrapper[2584]: E0917 23:10:26.759256    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-24tsm" podUID="fa08be4f-e950-4dee-a227-feb39c2ba7e0"
Sep 17 23:10:26.759567 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:26.759323    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4bbdp"
Sep 17 23:10:26.759567 ip-10-0-142-4 kubenswrapper[2584]: E0917 23:10:26.759435    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4bbdp" podUID="d672140f-ad59-4a68-9dde-80b62492af7f"
Sep 17 23:10:26.759567 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:26.759126    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wgps9"
Sep 17 23:10:26.759567 ip-10-0-142-4 kubenswrapper[2584]: E0917 23:10:26.759523    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wgps9" podUID="a2a26114-4c24-4dce-929c-3cb5caf17e6a"
Sep 17 23:10:28.337780 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:28.337705    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fa08be4f-e950-4dee-a227-feb39c2ba7e0-original-pull-secret\") pod \"global-pull-secret-syncer-24tsm\" (UID: \"fa08be4f-e950-4dee-a227-feb39c2ba7e0\") " pod="kube-system/global-pull-secret-syncer-24tsm"
Sep 17 23:10:28.338243 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:28.337812    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d672140f-ad59-4a68-9dde-80b62492af7f-metrics-certs\") pod \"network-metrics-daemon-4bbdp\" (UID: \"d672140f-ad59-4a68-9dde-80b62492af7f\") " pod="openshift-multus/network-metrics-daemon-4bbdp"
Sep 17 23:10:28.338243 ip-10-0-142-4 kubenswrapper[2584]: E0917 23:10:28.337903    2584 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 23:10:28.338243 ip-10-0-142-4 kubenswrapper[2584]: E0917 23:10:28.337976    2584 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 23:10:28.338243 ip-10-0-142-4 kubenswrapper[2584]: E0917 23:10:28.337979    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fa08be4f-e950-4dee-a227-feb39c2ba7e0-original-pull-secret podName:fa08be4f-e950-4dee-a227-feb39c2ba7e0 nodeName:}" failed. No retries permitted until 2026-09-17 23:10:36.337958405 +0000 UTC m=+19.703618302 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fa08be4f-e950-4dee-a227-feb39c2ba7e0-original-pull-secret") pod "global-pull-secret-syncer-24tsm" (UID: "fa08be4f-e950-4dee-a227-feb39c2ba7e0") : object "kube-system"/"original-pull-secret" not registered
Sep 17 23:10:28.338243 ip-10-0-142-4 kubenswrapper[2584]: E0917 23:10:28.338055    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d672140f-ad59-4a68-9dde-80b62492af7f-metrics-certs podName:d672140f-ad59-4a68-9dde-80b62492af7f nodeName:}" failed. No retries permitted until 2026-09-17 23:10:36.338037656 +0000 UTC m=+19.703697553 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d672140f-ad59-4a68-9dde-80b62492af7f-metrics-certs") pod "network-metrics-daemon-4bbdp" (UID: "d672140f-ad59-4a68-9dde-80b62492af7f") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 23:10:28.438946 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:28.438906    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bgsh5\" (UniqueName: \"kubernetes.io/projected/a2a26114-4c24-4dce-929c-3cb5caf17e6a-kube-api-access-bgsh5\") pod \"network-check-target-wgps9\" (UID: \"a2a26114-4c24-4dce-929c-3cb5caf17e6a\") " pod="openshift-network-diagnostics/network-check-target-wgps9"
Sep 17 23:10:28.439184 ip-10-0-142-4 kubenswrapper[2584]: E0917 23:10:28.439116    2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 23:10:28.439184 ip-10-0-142-4 kubenswrapper[2584]: E0917 23:10:28.439138    2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 23:10:28.439184 ip-10-0-142-4 kubenswrapper[2584]: E0917 23:10:28.439151    2584 projected.go:196] Error preparing data for projected volume kube-api-access-bgsh5 for pod openshift-network-diagnostics/network-check-target-wgps9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 23:10:28.439313 ip-10-0-142-4 kubenswrapper[2584]: E0917 23:10:28.439218    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a2a26114-4c24-4dce-929c-3cb5caf17e6a-kube-api-access-bgsh5 podName:a2a26114-4c24-4dce-929c-3cb5caf17e6a nodeName:}" failed. No retries permitted until 2026-09-17 23:10:36.43919967 +0000 UTC m=+19.804859569 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-bgsh5" (UniqueName: "kubernetes.io/projected/a2a26114-4c24-4dce-929c-3cb5caf17e6a-kube-api-access-bgsh5") pod "network-check-target-wgps9" (UID: "a2a26114-4c24-4dce-929c-3cb5caf17e6a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 23:10:28.759098 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:28.759070    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wgps9"
Sep 17 23:10:28.759273 ip-10-0-142-4 kubenswrapper[2584]: E0917 23:10:28.759149    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wgps9" podUID="a2a26114-4c24-4dce-929c-3cb5caf17e6a"
Sep 17 23:10:28.759418 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:28.759405    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-24tsm"
Sep 17 23:10:28.759479 ip-10-0-142-4 kubenswrapper[2584]: E0917 23:10:28.759465    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-24tsm" podUID="fa08be4f-e950-4dee-a227-feb39c2ba7e0"
Sep 17 23:10:28.759620 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:28.759610    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4bbdp"
Sep 17 23:10:28.759677 ip-10-0-142-4 kubenswrapper[2584]: E0917 23:10:28.759666    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4bbdp" podUID="d672140f-ad59-4a68-9dde-80b62492af7f"
Sep 17 23:10:30.758590 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:30.758550    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-24tsm"
Sep 17 23:10:30.758590 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:30.758572    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4bbdp"
Sep 17 23:10:30.759090 ip-10-0-142-4 kubenswrapper[2584]: E0917 23:10:30.758672    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-24tsm" podUID="fa08be4f-e950-4dee-a227-feb39c2ba7e0"
Sep 17 23:10:30.759090 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:30.758722    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wgps9"
Sep 17 23:10:30.759090 ip-10-0-142-4 kubenswrapper[2584]: E0917 23:10:30.758889    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wgps9" podUID="a2a26114-4c24-4dce-929c-3cb5caf17e6a"
Sep 17 23:10:30.759090 ip-10-0-142-4 kubenswrapper[2584]: E0917 23:10:30.759031    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4bbdp" podUID="d672140f-ad59-4a68-9dde-80b62492af7f"
Sep 17 23:10:32.758479 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:32.758447    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wgps9"
Sep 17 23:10:32.758936 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:32.758458    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-24tsm"
Sep 17 23:10:32.758936 ip-10-0-142-4 kubenswrapper[2584]: E0917 23:10:32.758566    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wgps9" podUID="a2a26114-4c24-4dce-929c-3cb5caf17e6a"
Sep 17 23:10:32.758936 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:32.758583    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4bbdp"
Sep 17 23:10:32.758936 ip-10-0-142-4 kubenswrapper[2584]: E0917 23:10:32.758708    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-24tsm" podUID="fa08be4f-e950-4dee-a227-feb39c2ba7e0"
Sep 17 23:10:32.758936 ip-10-0-142-4 kubenswrapper[2584]: E0917 23:10:32.758809    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4bbdp" podUID="d672140f-ad59-4a68-9dde-80b62492af7f"
Sep 17 23:10:34.757895 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:34.757767    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wgps9"
Sep 17 23:10:34.757895 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:34.757776    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-24tsm"
Sep 17 23:10:34.758430 ip-10-0-142-4 kubenswrapper[2584]: E0917 23:10:34.757943    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-24tsm" podUID="fa08be4f-e950-4dee-a227-feb39c2ba7e0"
Sep 17 23:10:34.758430 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:34.758300    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4bbdp"
Sep 17 23:10:34.758430 ip-10-0-142-4 kubenswrapper[2584]: E0917 23:10:34.758410    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4bbdp" podUID="d672140f-ad59-4a68-9dde-80b62492af7f"
Sep 17 23:10:34.758585 ip-10-0-142-4 kubenswrapper[2584]: E0917 23:10:34.758479    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wgps9" podUID="a2a26114-4c24-4dce-929c-3cb5caf17e6a"
Sep 17 23:10:36.398775 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:36.398733    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fa08be4f-e950-4dee-a227-feb39c2ba7e0-original-pull-secret\") pod \"global-pull-secret-syncer-24tsm\" (UID: \"fa08be4f-e950-4dee-a227-feb39c2ba7e0\") " pod="kube-system/global-pull-secret-syncer-24tsm"
Sep 17 23:10:36.399242 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:36.398800    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d672140f-ad59-4a68-9dde-80b62492af7f-metrics-certs\") pod \"network-metrics-daemon-4bbdp\" (UID: \"d672140f-ad59-4a68-9dde-80b62492af7f\") " pod="openshift-multus/network-metrics-daemon-4bbdp"
Sep 17 23:10:36.399242 ip-10-0-142-4 kubenswrapper[2584]: E0917 23:10:36.398919    2584 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 23:10:36.399242 ip-10-0-142-4 kubenswrapper[2584]: E0917 23:10:36.398977    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d672140f-ad59-4a68-9dde-80b62492af7f-metrics-certs podName:d672140f-ad59-4a68-9dde-80b62492af7f nodeName:}" failed. No retries permitted until 2026-09-17 23:10:52.398963594 +0000 UTC m=+35.764623489 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d672140f-ad59-4a68-9dde-80b62492af7f-metrics-certs") pod "network-metrics-daemon-4bbdp" (UID: "d672140f-ad59-4a68-9dde-80b62492af7f") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 23:10:36.399390 ip-10-0-142-4 kubenswrapper[2584]: E0917 23:10:36.399354    2584 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 23:10:36.399438 ip-10-0-142-4 kubenswrapper[2584]: E0917 23:10:36.399396    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fa08be4f-e950-4dee-a227-feb39c2ba7e0-original-pull-secret podName:fa08be4f-e950-4dee-a227-feb39c2ba7e0 nodeName:}" failed. No retries permitted until 2026-09-17 23:10:52.399385138 +0000 UTC m=+35.765045032 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fa08be4f-e950-4dee-a227-feb39c2ba7e0-original-pull-secret") pod "global-pull-secret-syncer-24tsm" (UID: "fa08be4f-e950-4dee-a227-feb39c2ba7e0") : object "kube-system"/"original-pull-secret" not registered
Sep 17 23:10:36.499612 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:36.499585    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bgsh5\" (UniqueName: \"kubernetes.io/projected/a2a26114-4c24-4dce-929c-3cb5caf17e6a-kube-api-access-bgsh5\") pod \"network-check-target-wgps9\" (UID: \"a2a26114-4c24-4dce-929c-3cb5caf17e6a\") " pod="openshift-network-diagnostics/network-check-target-wgps9"
Sep 17 23:10:36.499755 ip-10-0-142-4 kubenswrapper[2584]: E0917 23:10:36.499728    2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 23:10:36.499755 ip-10-0-142-4 kubenswrapper[2584]: E0917 23:10:36.499747    2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 23:10:36.499860 ip-10-0-142-4 kubenswrapper[2584]: E0917 23:10:36.499758    2584 projected.go:196] Error preparing data for projected volume kube-api-access-bgsh5 for pod openshift-network-diagnostics/network-check-target-wgps9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 23:10:36.499860 ip-10-0-142-4 kubenswrapper[2584]: E0917 23:10:36.499806    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a2a26114-4c24-4dce-929c-3cb5caf17e6a-kube-api-access-bgsh5 podName:a2a26114-4c24-4dce-929c-3cb5caf17e6a nodeName:}" failed. No retries permitted until 2026-09-17 23:10:52.499791336 +0000 UTC m=+35.865451231 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-bgsh5" (UniqueName: "kubernetes.io/projected/a2a26114-4c24-4dce-929c-3cb5caf17e6a-kube-api-access-bgsh5") pod "network-check-target-wgps9" (UID: "a2a26114-4c24-4dce-929c-3cb5caf17e6a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 23:10:36.757722 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:36.757687    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-24tsm"
Sep 17 23:10:36.757722 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:36.757720    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4bbdp"
Sep 17 23:10:36.757982 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:36.757730    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wgps9"
Sep 17 23:10:36.757982 ip-10-0-142-4 kubenswrapper[2584]: E0917 23:10:36.757877    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-24tsm" podUID="fa08be4f-e950-4dee-a227-feb39c2ba7e0"
Sep 17 23:10:36.757982 ip-10-0-142-4 kubenswrapper[2584]: E0917 23:10:36.757879    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wgps9" podUID="a2a26114-4c24-4dce-929c-3cb5caf17e6a"
Sep 17 23:10:36.757982 ip-10-0-142-4 kubenswrapper[2584]: E0917 23:10:36.757958    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4bbdp" podUID="d672140f-ad59-4a68-9dde-80b62492af7f"
Sep 17 23:10:38.758839 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:38.758802    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-24tsm"
Sep 17 23:10:38.759225 ip-10-0-142-4 kubenswrapper[2584]: E0917 23:10:38.758901    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-24tsm" podUID="fa08be4f-e950-4dee-a227-feb39c2ba7e0"
Sep 17 23:10:38.759225 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:38.759000    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4bbdp"
Sep 17 23:10:38.759225 ip-10-0-142-4 kubenswrapper[2584]: E0917 23:10:38.759125    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4bbdp" podUID="d672140f-ad59-4a68-9dde-80b62492af7f"
Sep 17 23:10:38.759225 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:38.759156    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wgps9"
Sep 17 23:10:38.759225 ip-10-0-142-4 kubenswrapper[2584]: E0917 23:10:38.759219    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wgps9" podUID="a2a26114-4c24-4dce-929c-3cb5caf17e6a"
Sep 17 23:10:39.997622 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:39.997239    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-jf4g9" event={"ID":"891302e3-91c1-4b56-a169-950bb8587aa5","Type":"ContainerStarted","Data":"cfeed84096c8cfe3c3868c3d950d923062f8608741e65d6363a64f0723de41bb"}
Sep 17 23:10:39.998724 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:39.998702    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-9j2wk" event={"ID":"1c71bd0e-1761-4157-82d0-a33701aece84","Type":"ContainerStarted","Data":"8b19a7aabc61bc597cde66a74c21198580f6a85a5cb507894176f362e77a9ead"}
Sep 17 23:10:40.001791 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:40.001769    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tcglq" event={"ID":"123417a3-f884-4c2f-b7ac-837813412743","Type":"ContainerStarted","Data":"f75ef8a82dcf5eb4028eaab8843a5022f9d45f2459249abc9693f228c10badac"}
Sep 17 23:10:40.001911 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:40.001862    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tcglq" event={"ID":"123417a3-f884-4c2f-b7ac-837813412743","Type":"ContainerStarted","Data":"824ce4be5282f511d308f30b4de2cab197788c36210ec17a3dcc0e5e10a657d2"}
Sep 17 23:10:40.001956 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:40.001918    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tcglq" event={"ID":"123417a3-f884-4c2f-b7ac-837813412743","Type":"ContainerStarted","Data":"8e3e1fbecc0e92bf99af101ef7ba975e00d537c941d870e89d5ff7ddf2c807f3"}
Sep 17 23:10:40.001998 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:40.001955    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tcglq" event={"ID":"123417a3-f884-4c2f-b7ac-837813412743","Type":"ContainerStarted","Data":"7d60bed72653f7af1275e2283ce46c28940af37080758c67fd371b2062457064"}
Sep 17 23:10:40.001998 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:40.001992    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tcglq" event={"ID":"123417a3-f884-4c2f-b7ac-837813412743","Type":"ContainerStarted","Data":"580e6b7d9f5d734e97955b82ad9d42fe63a56dd306a2d81968f89faa610c7860"}
Sep 17 23:10:40.002119 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:40.002056    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tcglq" event={"ID":"123417a3-f884-4c2f-b7ac-837813412743","Type":"ContainerStarted","Data":"c1771e8768397c64a624062e2651a741983ae9dc9b0b28374d7c868e8319ee60"}
Sep 17 23:10:40.003400 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:40.003380    2584 generic.go:356] "Generic (PLEG): container finished" podID="ea8e5a0a-14f1-4be6-84e1-33567e5d6032" containerID="d7081fdcfc721e06cddfb0a446784b0d3d4b84784f9f7b75e503a25779f2163e" exitCode=0
Sep 17 23:10:40.003476 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:40.003443    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2gjg8" event={"ID":"ea8e5a0a-14f1-4be6-84e1-33567e5d6032","Type":"ContainerDied","Data":"d7081fdcfc721e06cddfb0a446784b0d3d4b84784f9f7b75e503a25779f2163e"}
Sep 17 23:10:40.004930 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:40.004902    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-895t6" event={"ID":"fcc7ae4e-ace4-490f-a4e0-1c7c0ea22f2d","Type":"ContainerStarted","Data":"f6603e9259dd67cf1b257c8ef599c1b4b0af995d8f313642b6457648cc064356"}
Sep 17 23:10:40.006481 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:40.006456    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jph2d" event={"ID":"1ae5e3b8-3cf3-4664-b502-8ab797838fb8","Type":"ContainerStarted","Data":"a6845b957cd27975979510d82e8a2db5a512afae6424c76a78842bb42a7a3d8d"}
Sep 17 23:10:40.007783 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:40.007764    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-5spz7" event={"ID":"04f96656-0988-4bea-a70f-79d1b8445201","Type":"ContainerStarted","Data":"3b10dc64c5c5140216f5e45e36f8cdb565cfbcaad50307548ce258ff054a6d5a"}
Sep 17 23:10:40.009150 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:40.009128    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-g2vzq" event={"ID":"05765151-c799-4fe2-9f2e-cf4181e84ffa","Type":"ContainerStarted","Data":"3b429151d11fe03151cd4269949d28541d6e2504cb82771bd77f0dd732dc0e58"}
Sep 17 23:10:40.016142 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:40.016109    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-jf4g9" podStartSLOduration=4.495986036 podStartE2EDuration="22.01609805s" totalImagesPullingTime="17.520112014s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 23:10:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:10:40.01597551 +0000 UTC m=+23.381635425" watchObservedRunningTime="2026-09-17 23:10:40.01609805 +0000 UTC m=+23.381757965"
Sep 17 23:10:40.016521 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:40.016500    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-4.ec2.internal" podStartSLOduration=21.01649579 podStartE2EDuration="21.01649579s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 23:10:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:10:23.984054962 +0000 UTC m=+7.349714878" watchObservedRunningTime="2026-09-17 23:10:40.01649579 +0000 UTC m=+23.382155725"
Sep 17 23:10:40.037596 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:40.037547    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-g2vzq" podStartSLOduration=10.925071817 podStartE2EDuration="21.037529036s" totalImagesPullingTime="10.112457219s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:10:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:10:40.03707514 +0000 UTC m=+23.402735057" watchObservedRunningTime="2026-09-17 23:10:40.037529036 +0000 UTC m=+23.403188955"
Sep 17 23:10:40.139148 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:40.139096    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-895t6" podStartSLOduration=3.613016532 podStartE2EDuration="21.139077718s" totalImagesPullingTime="17.526061186s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 23:10:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:10:40.138855372 +0000 UTC m=+23.504515289" watchObservedRunningTime="2026-09-17 23:10:40.139077718 +0000 UTC m=+23.504737634"
Sep 17 23:10:40.139292 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:40.139262    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-9j2wk" podStartSLOduration=4.545578099 podStartE2EDuration="22.139255462s" totalImagesPullingTime="17.593677363s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 23:10:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:10:40.117227408 +0000 UTC m=+23.482887324" watchObservedRunningTime="2026-09-17 23:10:40.139255462 +0000 UTC m=+23.504915381"
Sep 17 23:10:40.171996 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:40.171946    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-5spz7" podStartSLOduration=3.652986211 podStartE2EDuration="21.171929091s" totalImagesPullingTime="17.51894288s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 23:10:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:10:40.171555974 +0000 UTC m=+23.537215890" watchObservedRunningTime="2026-09-17 23:10:40.171929091 +0000 UTC m=+23.537589008"
Sep 17 23:10:40.503996 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:40.503971    2584 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 17 23:10:40.727999 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:40.727868    2584 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-17T23:10:40.503993365Z","UUID":"42bbbbea-adb8-4606-8284-702fc086f7f7","Handler":null,"Name":"","Endpoint":""}
Sep 17 23:10:40.729411 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:40.729387    2584 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 17 23:10:40.729411 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:40.729414    2584 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 17 23:10:40.758649 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:40.758611    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-24tsm"
Sep 17 23:10:40.758784 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:40.758709    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4bbdp"
Sep 17 23:10:40.758784 ip-10-0-142-4 kubenswrapper[2584]: E0917 23:10:40.758720    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-24tsm" podUID="fa08be4f-e950-4dee-a227-feb39c2ba7e0"
Sep 17 23:10:40.758884 ip-10-0-142-4 kubenswrapper[2584]: E0917 23:10:40.758791    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4bbdp" podUID="d672140f-ad59-4a68-9dde-80b62492af7f"
Sep 17 23:10:40.758884 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:40.758836    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wgps9"
Sep 17 23:10:40.758986 ip-10-0-142-4 kubenswrapper[2584]: E0917 23:10:40.758905    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wgps9" podUID="a2a26114-4c24-4dce-929c-3cb5caf17e6a"
Sep 17 23:10:41.012470 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:41.012436    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-4bls7" event={"ID":"f6e69d33-8d35-4bbd-b113-e16f0b87f42c","Type":"ContainerStarted","Data":"f0149e2632471e09e90cc31b1c9a13489b8830e2813ce10fcac215830fa36213"}
Sep 17 23:10:41.014259 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:41.014231    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jph2d" event={"ID":"1ae5e3b8-3cf3-4664-b502-8ab797838fb8","Type":"ContainerStarted","Data":"747952872c605d6d6c29d6c4859b9ff24926868e29de9d75da611dc967e2b3e1"}
Sep 17 23:10:41.056552 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:41.056508    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-4bls7" podStartSLOduration=4.530248598 podStartE2EDuration="22.05649484s" totalImagesPullingTime="17.526246242s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 23:10:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:10:41.05642409 +0000 UTC m=+24.422084019" watchObservedRunningTime="2026-09-17 23:10:41.05649484 +0000 UTC m=+24.422154755"
Sep 17 23:10:42.020848 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:42.020806    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tcglq" event={"ID":"123417a3-f884-4c2f-b7ac-837813412743","Type":"ContainerStarted","Data":"c44014d216ceaef48dceb009d77fc45605cec15ca7731b003d4204c1973ff1bb"}
Sep 17 23:10:42.023238 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:42.023207    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jph2d" event={"ID":"1ae5e3b8-3cf3-4664-b502-8ab797838fb8","Type":"ContainerStarted","Data":"534808f4d7258f2665276d1732a06ac215654ec168f1aed790c1c926182d8846"}
Sep 17 23:10:42.757991 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:42.757961    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-24tsm"
Sep 17 23:10:42.758189 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:42.757961    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wgps9"
Sep 17 23:10:42.758189 ip-10-0-142-4 kubenswrapper[2584]: E0917 23:10:42.758078    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-24tsm" podUID="fa08be4f-e950-4dee-a227-feb39c2ba7e0"
Sep 17 23:10:42.758189 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:42.758086    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4bbdp"
Sep 17 23:10:42.758189 ip-10-0-142-4 kubenswrapper[2584]: E0917 23:10:42.758158    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wgps9" podUID="a2a26114-4c24-4dce-929c-3cb5caf17e6a"
Sep 17 23:10:42.758411 ip-10-0-142-4 kubenswrapper[2584]: E0917 23:10:42.758229    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4bbdp" podUID="d672140f-ad59-4a68-9dde-80b62492af7f"
Sep 17 23:10:44.507962 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:44.507785    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-g2vzq"
Sep 17 23:10:44.757201 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:44.757169    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-g2vzq"
Sep 17 23:10:44.757647 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:44.757631    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-24tsm"
Sep 17 23:10:44.757739 ip-10-0-142-4 kubenswrapper[2584]: E0917 23:10:44.757717    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-24tsm" podUID="fa08be4f-e950-4dee-a227-feb39c2ba7e0"
Sep 17 23:10:44.757953 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:44.757749    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4bbdp"
Sep 17 23:10:44.757953 ip-10-0-142-4 kubenswrapper[2584]: E0917 23:10:44.757865    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4bbdp" podUID="d672140f-ad59-4a68-9dde-80b62492af7f"
Sep 17 23:10:44.757953 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:44.757941    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wgps9"
Sep 17 23:10:44.758161 ip-10-0-142-4 kubenswrapper[2584]: E0917 23:10:44.758025    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wgps9" podUID="a2a26114-4c24-4dce-929c-3cb5caf17e6a"
Sep 17 23:10:44.760904 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:44.760886    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-g2vzq"
Sep 17 23:10:44.779537 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:44.779472    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jph2d" podStartSLOduration=6.398216348 podStartE2EDuration="25.779453576s" totalImagesPullingTime="19.381237228s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 23:10:19 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:10:42.060467933 +0000 UTC m=+25.426127849" watchObservedRunningTime="2026-09-17 23:10:44.779453576 +0000 UTC m=+28.145113492"
Sep 17 23:10:45.030207 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:45.029255    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-g2vzq"
Sep 17 23:10:46.033896 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:46.033865    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tcglq" event={"ID":"123417a3-f884-4c2f-b7ac-837813412743","Type":"ContainerStarted","Data":"8c62629136bee30dc380f632631260fdfd884a3c3a9d87010f814b7a4466697b"}
Sep 17 23:10:46.034701 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:46.034166    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-tcglq"
Sep 17 23:10:46.034701 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:46.034254    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-tcglq"
Sep 17 23:10:46.035870 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:46.035848    2584 generic.go:356] "Generic (PLEG): container finished" podID="ea8e5a0a-14f1-4be6-84e1-33567e5d6032" containerID="98fada33f8f208448b7bd3f9094f6d85d62faffa02be54ea7a1dec24e81c38dd" exitCode=0
Sep 17 23:10:46.035972 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:46.035921    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2gjg8" event={"ID":"ea8e5a0a-14f1-4be6-84e1-33567e5d6032","Type":"ContainerDied","Data":"98fada33f8f208448b7bd3f9094f6d85d62faffa02be54ea7a1dec24e81c38dd"}
Sep 17 23:10:46.049598 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:46.049577    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-tcglq"
Sep 17 23:10:46.072387 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:46.072353    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-tcglq" podStartSLOduration=10.548716219 podStartE2EDuration="28.072340305s" totalImagesPullingTime="17.523624086s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 23:10:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:10:46.071795886 +0000 UTC m=+29.437455814" watchObservedRunningTime="2026-09-17 23:10:46.072340305 +0000 UTC m=+29.438000221"
Sep 17 23:10:46.758089 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:46.757985    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-24tsm"
Sep 17 23:10:46.758089 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:46.758051    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4bbdp"
Sep 17 23:10:46.758914 ip-10-0-142-4 kubenswrapper[2584]: E0917 23:10:46.758679    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4bbdp" podUID="d672140f-ad59-4a68-9dde-80b62492af7f"
Sep 17 23:10:46.758914 ip-10-0-142-4 kubenswrapper[2584]: E0917 23:10:46.758130    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-24tsm" podUID="fa08be4f-e950-4dee-a227-feb39c2ba7e0"
Sep 17 23:10:46.758914 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:46.758746    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wgps9"
Sep 17 23:10:46.759169 ip-10-0-142-4 kubenswrapper[2584]: E0917 23:10:46.759052    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wgps9" podUID="a2a26114-4c24-4dce-929c-3cb5caf17e6a"
Sep 17 23:10:47.038425 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:47.038202    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-tcglq"
Sep 17 23:10:47.052760 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:47.052739    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-tcglq"
Sep 17 23:10:47.256714 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:47.256682    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-24tsm"]
Sep 17 23:10:47.256799 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:47.256788    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-24tsm"
Sep 17 23:10:47.256882 ip-10-0-142-4 kubenswrapper[2584]: E0917 23:10:47.256865    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-24tsm" podUID="fa08be4f-e950-4dee-a227-feb39c2ba7e0"
Sep 17 23:10:47.279722 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:47.279698    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-wgps9"]
Sep 17 23:10:47.279843 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:47.279808    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wgps9"
Sep 17 23:10:47.279913 ip-10-0-142-4 kubenswrapper[2584]: E0917 23:10:47.279896    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wgps9" podUID="a2a26114-4c24-4dce-929c-3cb5caf17e6a"
Sep 17 23:10:47.280257 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:47.280236    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-4bbdp"]
Sep 17 23:10:47.280348 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:47.280333    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4bbdp"
Sep 17 23:10:47.280428 ip-10-0-142-4 kubenswrapper[2584]: E0917 23:10:47.280412    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4bbdp" podUID="d672140f-ad59-4a68-9dde-80b62492af7f"
Sep 17 23:10:48.042220 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:48.042190    2584 generic.go:356] "Generic (PLEG): container finished" podID="ea8e5a0a-14f1-4be6-84e1-33567e5d6032" containerID="d40ab0e974455bc07fe2b6d3cbaac63977f9de84683bdbad1ad45848d0b22594" exitCode=0
Sep 17 23:10:48.042584 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:48.042274    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2gjg8" event={"ID":"ea8e5a0a-14f1-4be6-84e1-33567e5d6032","Type":"ContainerDied","Data":"d40ab0e974455bc07fe2b6d3cbaac63977f9de84683bdbad1ad45848d0b22594"}
Sep 17 23:10:48.764522 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:48.764491    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4bbdp"
Sep 17 23:10:48.764522 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:48.764509    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wgps9"
Sep 17 23:10:48.764747 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:48.764597    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-24tsm"
Sep 17 23:10:48.764747 ip-10-0-142-4 kubenswrapper[2584]: E0917 23:10:48.764611    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4bbdp" podUID="d672140f-ad59-4a68-9dde-80b62492af7f"
Sep 17 23:10:48.764747 ip-10-0-142-4 kubenswrapper[2584]: E0917 23:10:48.764701    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-24tsm" podUID="fa08be4f-e950-4dee-a227-feb39c2ba7e0"
Sep 17 23:10:48.764904 ip-10-0-142-4 kubenswrapper[2584]: E0917 23:10:48.764746    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wgps9" podUID="a2a26114-4c24-4dce-929c-3cb5caf17e6a"
Sep 17 23:10:50.048332 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:50.048303    2584 generic.go:356] "Generic (PLEG): container finished" podID="ea8e5a0a-14f1-4be6-84e1-33567e5d6032" containerID="bbc7131da19779cd50fd6873d9a3a86e384c733d782ebdffb17e43b3a1d06a41" exitCode=0
Sep 17 23:10:50.048633 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:50.048345    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2gjg8" event={"ID":"ea8e5a0a-14f1-4be6-84e1-33567e5d6032","Type":"ContainerDied","Data":"bbc7131da19779cd50fd6873d9a3a86e384c733d782ebdffb17e43b3a1d06a41"}
Sep 17 23:10:50.761565 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:50.761538    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wgps9"
Sep 17 23:10:50.761758 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:50.761636    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-24tsm"
Sep 17 23:10:50.761758 ip-10-0-142-4 kubenswrapper[2584]: E0917 23:10:50.761651    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wgps9" podUID="a2a26114-4c24-4dce-929c-3cb5caf17e6a"
Sep 17 23:10:50.761758 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:50.761670    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4bbdp"
Sep 17 23:10:50.761758 ip-10-0-142-4 kubenswrapper[2584]: E0917 23:10:50.761728    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-24tsm" podUID="fa08be4f-e950-4dee-a227-feb39c2ba7e0"
Sep 17 23:10:50.762035 ip-10-0-142-4 kubenswrapper[2584]: E0917 23:10:50.761821    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4bbdp" podUID="d672140f-ad59-4a68-9dde-80b62492af7f"
Sep 17 23:10:52.022461 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:52.022436    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-4.ec2.internal" event="NodeReady"
Sep 17 23:10:52.022872 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:52.022571    2584 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 17 23:10:52.065626 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:52.065592    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-75f6fdbdff-5rgvh"]
Sep 17 23:10:52.084841 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:52.084786    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-75f6fdbdff-5rgvh"
Sep 17 23:10:52.087815 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:52.087791    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-bkxmq"]
Sep 17 23:10:52.090073 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:52.089841    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 17 23:10:52.090073 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:52.089899    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 17 23:10:52.090073 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:52.089910    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-5987r\""
Sep 17 23:10:52.098755 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:52.098735    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 17 23:10:52.100116 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:52.100081    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 17 23:10:52.108348 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:52.108303    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-qtz2c"]
Sep 17 23:10:52.108474 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:52.108458    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-bkxmq"
Sep 17 23:10:52.111317 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:52.111140    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 17 23:10:52.111317 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:52.111203    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-nkbz8\""
Sep 17 23:10:52.111317 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:52.111239    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 17 23:10:52.114129 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:52.114109    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 17 23:10:52.124744 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:52.124725    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-qtz2c"
Sep 17 23:10:52.124744 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:52.124728    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-75f6fdbdff-5rgvh"]
Sep 17 23:10:52.124876 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:52.124780    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-qtz2c"]
Sep 17 23:10:52.124876 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:52.124803    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-bkxmq"]
Sep 17 23:10:52.127327 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:52.127051    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 17 23:10:52.127327 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:52.127083    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-42llv\""
Sep 17 23:10:52.127327 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:52.127208    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 17 23:10:52.214572 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:52.214542    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9572448f-49db-4e3b-a174-f994660e7ef1-metrics-tls\") pod \"dns-default-qtz2c\" (UID: \"9572448f-49db-4e3b-a174-f994660e7ef1\") " pod="openshift-dns/dns-default-qtz2c"
Sep 17 23:10:52.214572 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:52.214568    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/35051393-90ef-4a48-b95c-a2cc221b538f-cert\") pod \"ingress-canary-bkxmq\" (UID: \"35051393-90ef-4a48-b95c-a2cc221b538f\") " pod="openshift-ingress-canary/ingress-canary-bkxmq"
Sep 17 23:10:52.214747 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:52.214594    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/06535038-f186-42c3-b9d1-707e2c7ea285-trusted-ca\") pod \"image-registry-75f6fdbdff-5rgvh\" (UID: \"06535038-f186-42c3-b9d1-707e2c7ea285\") " pod="openshift-image-registry/image-registry-75f6fdbdff-5rgvh"
Sep 17 23:10:52.214747 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:52.214621    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dj65p\" (UniqueName: \"kubernetes.io/projected/06535038-f186-42c3-b9d1-707e2c7ea285-kube-api-access-dj65p\") pod \"image-registry-75f6fdbdff-5rgvh\" (UID: \"06535038-f186-42c3-b9d1-707e2c7ea285\") " pod="openshift-image-registry/image-registry-75f6fdbdff-5rgvh"
Sep 17 23:10:52.214747 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:52.214685    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/06535038-f186-42c3-b9d1-707e2c7ea285-ca-trust-extracted\") pod \"image-registry-75f6fdbdff-5rgvh\" (UID: \"06535038-f186-42c3-b9d1-707e2c7ea285\") " pod="openshift-image-registry/image-registry-75f6fdbdff-5rgvh"
Sep 17 23:10:52.214747 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:52.214729    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/06535038-f186-42c3-b9d1-707e2c7ea285-bound-sa-token\") pod \"image-registry-75f6fdbdff-5rgvh\" (UID: \"06535038-f186-42c3-b9d1-707e2c7ea285\") " pod="openshift-image-registry/image-registry-75f6fdbdff-5rgvh"
Sep 17 23:10:52.214901 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:52.214762    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v79v9\" (UniqueName: \"kubernetes.io/projected/35051393-90ef-4a48-b95c-a2cc221b538f-kube-api-access-v79v9\") pod \"ingress-canary-bkxmq\" (UID: \"35051393-90ef-4a48-b95c-a2cc221b538f\") " pod="openshift-ingress-canary/ingress-canary-bkxmq"
Sep 17 23:10:52.214901 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:52.214814    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/06535038-f186-42c3-b9d1-707e2c7ea285-registry-certificates\") pod \"image-registry-75f6fdbdff-5rgvh\" (UID: \"06535038-f186-42c3-b9d1-707e2c7ea285\") " pod="openshift-image-registry/image-registry-75f6fdbdff-5rgvh"
Sep 17 23:10:52.214901 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:52.214859    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/9572448f-49db-4e3b-a174-f994660e7ef1-config-volume\") pod \"dns-default-qtz2c\" (UID: \"9572448f-49db-4e3b-a174-f994660e7ef1\") " pod="openshift-dns/dns-default-qtz2c"
Sep 17 23:10:52.215078 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:52.214918    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/06535038-f186-42c3-b9d1-707e2c7ea285-registry-tls\") pod \"image-registry-75f6fdbdff-5rgvh\" (UID: \"06535038-f186-42c3-b9d1-707e2c7ea285\") " pod="openshift-image-registry/image-registry-75f6fdbdff-5rgvh"
Sep 17 23:10:52.215078 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:52.214994    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/06535038-f186-42c3-b9d1-707e2c7ea285-installation-pull-secrets\") pod \"image-registry-75f6fdbdff-5rgvh\" (UID: \"06535038-f186-42c3-b9d1-707e2c7ea285\") " pod="openshift-image-registry/image-registry-75f6fdbdff-5rgvh"
Sep 17 23:10:52.215078 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:52.215061    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/06535038-f186-42c3-b9d1-707e2c7ea285-image-registry-private-configuration\") pod \"image-registry-75f6fdbdff-5rgvh\" (UID: \"06535038-f186-42c3-b9d1-707e2c7ea285\") " pod="openshift-image-registry/image-registry-75f6fdbdff-5rgvh"
Sep 17 23:10:52.215190 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:52.215090    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9572448f-49db-4e3b-a174-f994660e7ef1-tmp-dir\") pod \"dns-default-qtz2c\" (UID: \"9572448f-49db-4e3b-a174-f994660e7ef1\") " pod="openshift-dns/dns-default-qtz2c"
Sep 17 23:10:52.215190 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:52.215117    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nndq8\" (UniqueName: \"kubernetes.io/projected/9572448f-49db-4e3b-a174-f994660e7ef1-kube-api-access-nndq8\") pod \"dns-default-qtz2c\" (UID: \"9572448f-49db-4e3b-a174-f994660e7ef1\") " pod="openshift-dns/dns-default-qtz2c"
Sep 17 23:10:52.273395 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:52.273361    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:10:52.316400 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:52.316377    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9572448f-49db-4e3b-a174-f994660e7ef1-metrics-tls\") pod \"dns-default-qtz2c\" (UID: \"9572448f-49db-4e3b-a174-f994660e7ef1\") " pod="openshift-dns/dns-default-qtz2c"
Sep 17 23:10:52.316541 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:52.316409    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/35051393-90ef-4a48-b95c-a2cc221b538f-cert\") pod \"ingress-canary-bkxmq\" (UID: \"35051393-90ef-4a48-b95c-a2cc221b538f\") " pod="openshift-ingress-canary/ingress-canary-bkxmq"
Sep 17 23:10:52.316541 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:52.316433    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/06535038-f186-42c3-b9d1-707e2c7ea285-trusted-ca\") pod \"image-registry-75f6fdbdff-5rgvh\" (UID: \"06535038-f186-42c3-b9d1-707e2c7ea285\") " pod="openshift-image-registry/image-registry-75f6fdbdff-5rgvh"
Sep 17 23:10:52.316541 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:52.316454    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dj65p\" (UniqueName: \"kubernetes.io/projected/06535038-f186-42c3-b9d1-707e2c7ea285-kube-api-access-dj65p\") pod \"image-registry-75f6fdbdff-5rgvh\" (UID: \"06535038-f186-42c3-b9d1-707e2c7ea285\") " pod="openshift-image-registry/image-registry-75f6fdbdff-5rgvh"
Sep 17 23:10:52.316541 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:52.316486    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/06535038-f186-42c3-b9d1-707e2c7ea285-ca-trust-extracted\") pod \"image-registry-75f6fdbdff-5rgvh\" (UID: \"06535038-f186-42c3-b9d1-707e2c7ea285\") " pod="openshift-image-registry/image-registry-75f6fdbdff-5rgvh"
Sep 17 23:10:52.316541 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:52.316507    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/06535038-f186-42c3-b9d1-707e2c7ea285-bound-sa-token\") pod \"image-registry-75f6fdbdff-5rgvh\" (UID: \"06535038-f186-42c3-b9d1-707e2c7ea285\") " pod="openshift-image-registry/image-registry-75f6fdbdff-5rgvh"
Sep 17 23:10:52.316541 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:52.316530    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v79v9\" (UniqueName: \"kubernetes.io/projected/35051393-90ef-4a48-b95c-a2cc221b538f-kube-api-access-v79v9\") pod \"ingress-canary-bkxmq\" (UID: \"35051393-90ef-4a48-b95c-a2cc221b538f\") " pod="openshift-ingress-canary/ingress-canary-bkxmq"
Sep 17 23:10:52.316823 ip-10-0-142-4 kubenswrapper[2584]: E0917 23:10:52.316539    2584 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 23:10:52.316823 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:52.316557    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/06535038-f186-42c3-b9d1-707e2c7ea285-registry-certificates\") pod \"image-registry-75f6fdbdff-5rgvh\" (UID: \"06535038-f186-42c3-b9d1-707e2c7ea285\") " pod="openshift-image-registry/image-registry-75f6fdbdff-5rgvh"
Sep 17 23:10:52.316823 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:52.316584    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/9572448f-49db-4e3b-a174-f994660e7ef1-config-volume\") pod \"dns-default-qtz2c\" (UID: \"9572448f-49db-4e3b-a174-f994660e7ef1\") " pod="openshift-dns/dns-default-qtz2c"
Sep 17 23:10:52.316823 ip-10-0-142-4 kubenswrapper[2584]: E0917 23:10:52.316614    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9572448f-49db-4e3b-a174-f994660e7ef1-metrics-tls podName:9572448f-49db-4e3b-a174-f994660e7ef1 nodeName:}" failed. No retries permitted until 2026-09-17 23:10:52.816592024 +0000 UTC m=+36.182251923 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9572448f-49db-4e3b-a174-f994660e7ef1-metrics-tls") pod "dns-default-qtz2c" (UID: "9572448f-49db-4e3b-a174-f994660e7ef1") : secret "dns-default-metrics-tls" not found
Sep 17 23:10:52.316823 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:52.316654    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/06535038-f186-42c3-b9d1-707e2c7ea285-registry-tls\") pod \"image-registry-75f6fdbdff-5rgvh\" (UID: \"06535038-f186-42c3-b9d1-707e2c7ea285\") " pod="openshift-image-registry/image-registry-75f6fdbdff-5rgvh"
Sep 17 23:10:52.316823 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:52.316696    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/06535038-f186-42c3-b9d1-707e2c7ea285-installation-pull-secrets\") pod \"image-registry-75f6fdbdff-5rgvh\" (UID: \"06535038-f186-42c3-b9d1-707e2c7ea285\") " pod="openshift-image-registry/image-registry-75f6fdbdff-5rgvh"
Sep 17 23:10:52.316823 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:52.316728    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/06535038-f186-42c3-b9d1-707e2c7ea285-image-registry-private-configuration\") pod \"image-registry-75f6fdbdff-5rgvh\" (UID: \"06535038-f186-42c3-b9d1-707e2c7ea285\") " pod="openshift-image-registry/image-registry-75f6fdbdff-5rgvh"
Sep 17 23:10:52.316823 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:52.316754    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9572448f-49db-4e3b-a174-f994660e7ef1-tmp-dir\") pod \"dns-default-qtz2c\" (UID: \"9572448f-49db-4e3b-a174-f994660e7ef1\") " pod="openshift-dns/dns-default-qtz2c"
Sep 17 23:10:52.316823 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:52.316781    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nndq8\" (UniqueName: \"kubernetes.io/projected/9572448f-49db-4e3b-a174-f994660e7ef1-kube-api-access-nndq8\") pod \"dns-default-qtz2c\" (UID: \"9572448f-49db-4e3b-a174-f994660e7ef1\") " pod="openshift-dns/dns-default-qtz2c"
Sep 17 23:10:52.317318 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:52.316900    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/06535038-f186-42c3-b9d1-707e2c7ea285-ca-trust-extracted\") pod \"image-registry-75f6fdbdff-5rgvh\" (UID: \"06535038-f186-42c3-b9d1-707e2c7ea285\") " pod="openshift-image-registry/image-registry-75f6fdbdff-5rgvh"
Sep 17 23:10:52.317318 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:52.317231    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/9572448f-49db-4e3b-a174-f994660e7ef1-config-volume\") pod \"dns-default-qtz2c\" (UID: \"9572448f-49db-4e3b-a174-f994660e7ef1\") " pod="openshift-dns/dns-default-qtz2c"
Sep 17 23:10:52.317415 ip-10-0-142-4 kubenswrapper[2584]: E0917 23:10:52.317323    2584 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 23:10:52.317415 ip-10-0-142-4 kubenswrapper[2584]: E0917 23:10:52.317337    2584 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-75f6fdbdff-5rgvh: secret "image-registry-tls" not found
Sep 17 23:10:52.317415 ip-10-0-142-4 kubenswrapper[2584]: E0917 23:10:52.317394    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/06535038-f186-42c3-b9d1-707e2c7ea285-registry-tls podName:06535038-f186-42c3-b9d1-707e2c7ea285 nodeName:}" failed. No retries permitted until 2026-09-17 23:10:52.817377572 +0000 UTC m=+36.183037489 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/06535038-f186-42c3-b9d1-707e2c7ea285-registry-tls") pod "image-registry-75f6fdbdff-5rgvh" (UID: "06535038-f186-42c3-b9d1-707e2c7ea285") : secret "image-registry-tls" not found
Sep 17 23:10:52.317562 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:52.317442    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/06535038-f186-42c3-b9d1-707e2c7ea285-trusted-ca\") pod \"image-registry-75f6fdbdff-5rgvh\" (UID: \"06535038-f186-42c3-b9d1-707e2c7ea285\") " pod="openshift-image-registry/image-registry-75f6fdbdff-5rgvh"
Sep 17 23:10:52.317609 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:52.317562    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9572448f-49db-4e3b-a174-f994660e7ef1-tmp-dir\") pod \"dns-default-qtz2c\" (UID: \"9572448f-49db-4e3b-a174-f994660e7ef1\") " pod="openshift-dns/dns-default-qtz2c"
Sep 17 23:10:52.317717 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:52.317691    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/06535038-f186-42c3-b9d1-707e2c7ea285-registry-certificates\") pod \"image-registry-75f6fdbdff-5rgvh\" (UID: \"06535038-f186-42c3-b9d1-707e2c7ea285\") " pod="openshift-image-registry/image-registry-75f6fdbdff-5rgvh"
Sep 17 23:10:52.320956 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:52.320938    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/06535038-f186-42c3-b9d1-707e2c7ea285-installation-pull-secrets\") pod \"image-registry-75f6fdbdff-5rgvh\" (UID: \"06535038-f186-42c3-b9d1-707e2c7ea285\") " pod="openshift-image-registry/image-registry-75f6fdbdff-5rgvh"
Sep 17 23:10:52.320956 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:52.320951    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/06535038-f186-42c3-b9d1-707e2c7ea285-image-registry-private-configuration\") pod \"image-registry-75f6fdbdff-5rgvh\" (UID: \"06535038-f186-42c3-b9d1-707e2c7ea285\") " pod="openshift-image-registry/image-registry-75f6fdbdff-5rgvh"
Sep 17 23:10:52.321108 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:52.321031    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/35051393-90ef-4a48-b95c-a2cc221b538f-cert\") pod \"ingress-canary-bkxmq\" (UID: \"35051393-90ef-4a48-b95c-a2cc221b538f\") " pod="openshift-ingress-canary/ingress-canary-bkxmq"
Sep 17 23:10:52.328590 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:52.328566    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v79v9\" (UniqueName: \"kubernetes.io/projected/35051393-90ef-4a48-b95c-a2cc221b538f-kube-api-access-v79v9\") pod \"ingress-canary-bkxmq\" (UID: \"35051393-90ef-4a48-b95c-a2cc221b538f\") " pod="openshift-ingress-canary/ingress-canary-bkxmq"
Sep 17 23:10:52.328983 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:52.328962    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nndq8\" (UniqueName: \"kubernetes.io/projected/9572448f-49db-4e3b-a174-f994660e7ef1-kube-api-access-nndq8\") pod \"dns-default-qtz2c\" (UID: \"9572448f-49db-4e3b-a174-f994660e7ef1\") " pod="openshift-dns/dns-default-qtz2c"
Sep 17 23:10:52.329317 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:52.329277    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/06535038-f186-42c3-b9d1-707e2c7ea285-bound-sa-token\") pod \"image-registry-75f6fdbdff-5rgvh\" (UID: \"06535038-f186-42c3-b9d1-707e2c7ea285\") " pod="openshift-image-registry/image-registry-75f6fdbdff-5rgvh"
Sep 17 23:10:52.329706 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:52.329687    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dj65p\" (UniqueName: \"kubernetes.io/projected/06535038-f186-42c3-b9d1-707e2c7ea285-kube-api-access-dj65p\") pod \"image-registry-75f6fdbdff-5rgvh\" (UID: \"06535038-f186-42c3-b9d1-707e2c7ea285\") " pod="openshift-image-registry/image-registry-75f6fdbdff-5rgvh"
Sep 17 23:10:52.418000 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:52.417939    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fa08be4f-e950-4dee-a227-feb39c2ba7e0-original-pull-secret\") pod \"global-pull-secret-syncer-24tsm\" (UID: \"fa08be4f-e950-4dee-a227-feb39c2ba7e0\") " pod="kube-system/global-pull-secret-syncer-24tsm"
Sep 17 23:10:52.418126 ip-10-0-142-4 kubenswrapper[2584]: E0917 23:10:52.418094    2584 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 23:10:52.418126 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:52.418102    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d672140f-ad59-4a68-9dde-80b62492af7f-metrics-certs\") pod \"network-metrics-daemon-4bbdp\" (UID: \"d672140f-ad59-4a68-9dde-80b62492af7f\") " pod="openshift-multus/network-metrics-daemon-4bbdp"
Sep 17 23:10:52.418210 ip-10-0-142-4 kubenswrapper[2584]: E0917 23:10:52.418159    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fa08be4f-e950-4dee-a227-feb39c2ba7e0-original-pull-secret podName:fa08be4f-e950-4dee-a227-feb39c2ba7e0 nodeName:}" failed. No retries permitted until 2026-09-17 23:11:24.418142866 +0000 UTC m=+67.783802763 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fa08be4f-e950-4dee-a227-feb39c2ba7e0-original-pull-secret") pod "global-pull-secret-syncer-24tsm" (UID: "fa08be4f-e950-4dee-a227-feb39c2ba7e0") : object "kube-system"/"original-pull-secret" not registered
Sep 17 23:10:52.418210 ip-10-0-142-4 kubenswrapper[2584]: E0917 23:10:52.418197    2584 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 23:10:52.418299 ip-10-0-142-4 kubenswrapper[2584]: E0917 23:10:52.418237    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d672140f-ad59-4a68-9dde-80b62492af7f-metrics-certs podName:d672140f-ad59-4a68-9dde-80b62492af7f nodeName:}" failed. No retries permitted until 2026-09-17 23:11:24.41822603 +0000 UTC m=+67.783885927 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d672140f-ad59-4a68-9dde-80b62492af7f-metrics-certs") pod "network-metrics-daemon-4bbdp" (UID: "d672140f-ad59-4a68-9dde-80b62492af7f") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 23:10:52.420886 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:52.420863    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-bkxmq"
Sep 17 23:10:52.428331 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:52.428305    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:10:52.518503 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:52.518454    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bgsh5\" (UniqueName: \"kubernetes.io/projected/a2a26114-4c24-4dce-929c-3cb5caf17e6a-kube-api-access-bgsh5\") pod \"network-check-target-wgps9\" (UID: \"a2a26114-4c24-4dce-929c-3cb5caf17e6a\") " pod="openshift-network-diagnostics/network-check-target-wgps9"
Sep 17 23:10:52.518684 ip-10-0-142-4 kubenswrapper[2584]: E0917 23:10:52.518661    2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 23:10:52.518732 ip-10-0-142-4 kubenswrapper[2584]: E0917 23:10:52.518694    2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 23:10:52.518732 ip-10-0-142-4 kubenswrapper[2584]: E0917 23:10:52.518708    2584 projected.go:196] Error preparing data for projected volume kube-api-access-bgsh5 for pod openshift-network-diagnostics/network-check-target-wgps9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 23:10:52.518803 ip-10-0-142-4 kubenswrapper[2584]: E0917 23:10:52.518781    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a2a26114-4c24-4dce-929c-3cb5caf17e6a-kube-api-access-bgsh5 podName:a2a26114-4c24-4dce-929c-3cb5caf17e6a nodeName:}" failed. No retries permitted until 2026-09-17 23:11:24.518761179 +0000 UTC m=+67.884421093 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-bgsh5" (UniqueName: "kubernetes.io/projected/a2a26114-4c24-4dce-929c-3cb5caf17e6a-kube-api-access-bgsh5") pod "network-check-target-wgps9" (UID: "a2a26114-4c24-4dce-929c-3cb5caf17e6a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 23:10:52.735690 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:52.735623    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-bkxmq"]
Sep 17 23:10:52.739633 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:10:52.739607    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35051393_90ef_4a48_b95c_a2cc221b538f.slice/crio-71d1ee7480f2c6aa577998109c4946ecfe4ea39098a8ce05e57045d901619329 WatchSource:0}: Error finding container 71d1ee7480f2c6aa577998109c4946ecfe4ea39098a8ce05e57045d901619329: Status 404 returned error can't find the container with id 71d1ee7480f2c6aa577998109c4946ecfe4ea39098a8ce05e57045d901619329
Sep 17 23:10:52.758973 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:52.758438    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4bbdp"
Sep 17 23:10:52.758973 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:52.758453    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-24tsm"
Sep 17 23:10:52.758973 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:52.758673    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wgps9"
Sep 17 23:10:52.761185 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:52.761169    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 17 23:10:52.761282 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:52.761260    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-jknmn\""
Sep 17 23:10:52.763965 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:52.761596    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 17 23:10:52.763965 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:52.762742    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-k7845\""
Sep 17 23:10:52.763965 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:52.762860    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 17 23:10:52.763965 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:52.763242    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 17 23:10:52.821094 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:52.821073    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/06535038-f186-42c3-b9d1-707e2c7ea285-registry-tls\") pod \"image-registry-75f6fdbdff-5rgvh\" (UID: \"06535038-f186-42c3-b9d1-707e2c7ea285\") " pod="openshift-image-registry/image-registry-75f6fdbdff-5rgvh"
Sep 17 23:10:52.821212 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:52.821132    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9572448f-49db-4e3b-a174-f994660e7ef1-metrics-tls\") pod \"dns-default-qtz2c\" (UID: \"9572448f-49db-4e3b-a174-f994660e7ef1\") " pod="openshift-dns/dns-default-qtz2c"
Sep 17 23:10:52.821212 ip-10-0-142-4 kubenswrapper[2584]: E0917 23:10:52.821208    2584 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 23:10:52.821279 ip-10-0-142-4 kubenswrapper[2584]: E0917 23:10:52.821213    2584 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 23:10:52.821279 ip-10-0-142-4 kubenswrapper[2584]: E0917 23:10:52.821228    2584 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-75f6fdbdff-5rgvh: secret "image-registry-tls" not found
Sep 17 23:10:52.821279 ip-10-0-142-4 kubenswrapper[2584]: E0917 23:10:52.821255    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9572448f-49db-4e3b-a174-f994660e7ef1-metrics-tls podName:9572448f-49db-4e3b-a174-f994660e7ef1 nodeName:}" failed. No retries permitted until 2026-09-17 23:10:53.821242518 +0000 UTC m=+37.186902412 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9572448f-49db-4e3b-a174-f994660e7ef1-metrics-tls") pod "dns-default-qtz2c" (UID: "9572448f-49db-4e3b-a174-f994660e7ef1") : secret "dns-default-metrics-tls" not found
Sep 17 23:10:52.821279 ip-10-0-142-4 kubenswrapper[2584]: E0917 23:10:52.821273    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/06535038-f186-42c3-b9d1-707e2c7ea285-registry-tls podName:06535038-f186-42c3-b9d1-707e2c7ea285 nodeName:}" failed. No retries permitted until 2026-09-17 23:10:53.821261309 +0000 UTC m=+37.186921207 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/06535038-f186-42c3-b9d1-707e2c7ea285-registry-tls") pod "image-registry-75f6fdbdff-5rgvh" (UID: "06535038-f186-42c3-b9d1-707e2c7ea285") : secret "image-registry-tls" not found
Sep 17 23:10:52.913545 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:52.913524    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:10:53.058253 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:53.058220    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-bkxmq" event={"ID":"35051393-90ef-4a48-b95c-a2cc221b538f","Type":"ContainerStarted","Data":"71d1ee7480f2c6aa577998109c4946ecfe4ea39098a8ce05e57045d901619329"}
Sep 17 23:10:53.829251 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:53.829216    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9572448f-49db-4e3b-a174-f994660e7ef1-metrics-tls\") pod \"dns-default-qtz2c\" (UID: \"9572448f-49db-4e3b-a174-f994660e7ef1\") " pod="openshift-dns/dns-default-qtz2c"
Sep 17 23:10:53.829444 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:53.829272    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/06535038-f186-42c3-b9d1-707e2c7ea285-registry-tls\") pod \"image-registry-75f6fdbdff-5rgvh\" (UID: \"06535038-f186-42c3-b9d1-707e2c7ea285\") " pod="openshift-image-registry/image-registry-75f6fdbdff-5rgvh"
Sep 17 23:10:53.829444 ip-10-0-142-4 kubenswrapper[2584]: E0917 23:10:53.829375    2584 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 23:10:53.829444 ip-10-0-142-4 kubenswrapper[2584]: E0917 23:10:53.829385    2584 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 23:10:53.829444 ip-10-0-142-4 kubenswrapper[2584]: E0917 23:10:53.829404    2584 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-75f6fdbdff-5rgvh: secret "image-registry-tls" not found
Sep 17 23:10:53.829637 ip-10-0-142-4 kubenswrapper[2584]: E0917 23:10:53.829452    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9572448f-49db-4e3b-a174-f994660e7ef1-metrics-tls podName:9572448f-49db-4e3b-a174-f994660e7ef1 nodeName:}" failed. No retries permitted until 2026-09-17 23:10:55.829432012 +0000 UTC m=+39.195091924 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9572448f-49db-4e3b-a174-f994660e7ef1-metrics-tls") pod "dns-default-qtz2c" (UID: "9572448f-49db-4e3b-a174-f994660e7ef1") : secret "dns-default-metrics-tls" not found
Sep 17 23:10:53.829637 ip-10-0-142-4 kubenswrapper[2584]: E0917 23:10:53.829470    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/06535038-f186-42c3-b9d1-707e2c7ea285-registry-tls podName:06535038-f186-42c3-b9d1-707e2c7ea285 nodeName:}" failed. No retries permitted until 2026-09-17 23:10:55.829461551 +0000 UTC m=+39.195121445 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/06535038-f186-42c3-b9d1-707e2c7ea285-registry-tls") pod "image-registry-75f6fdbdff-5rgvh" (UID: "06535038-f186-42c3-b9d1-707e2c7ea285") : secret "image-registry-tls" not found
Sep 17 23:10:55.066901 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:55.066840    2584 generic.go:356] "Generic (PLEG): container finished" podID="ea8e5a0a-14f1-4be6-84e1-33567e5d6032" containerID="05410e4755c752fd96107d6bc9c1c4687ce1c183925dc3e23450215da0e22625" exitCode=0
Sep 17 23:10:55.066901 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:55.066890    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2gjg8" event={"ID":"ea8e5a0a-14f1-4be6-84e1-33567e5d6032","Type":"ContainerDied","Data":"05410e4755c752fd96107d6bc9c1c4687ce1c183925dc3e23450215da0e22625"}
Sep 17 23:10:55.847179 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:55.847145    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9572448f-49db-4e3b-a174-f994660e7ef1-metrics-tls\") pod \"dns-default-qtz2c\" (UID: \"9572448f-49db-4e3b-a174-f994660e7ef1\") " pod="openshift-dns/dns-default-qtz2c"
Sep 17 23:10:55.847319 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:55.847193    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/06535038-f186-42c3-b9d1-707e2c7ea285-registry-tls\") pod \"image-registry-75f6fdbdff-5rgvh\" (UID: \"06535038-f186-42c3-b9d1-707e2c7ea285\") " pod="openshift-image-registry/image-registry-75f6fdbdff-5rgvh"
Sep 17 23:10:55.847319 ip-10-0-142-4 kubenswrapper[2584]: E0917 23:10:55.847289    2584 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 23:10:55.847319 ip-10-0-142-4 kubenswrapper[2584]: E0917 23:10:55.847295    2584 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 23:10:55.847433 ip-10-0-142-4 kubenswrapper[2584]: E0917 23:10:55.847362    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9572448f-49db-4e3b-a174-f994660e7ef1-metrics-tls podName:9572448f-49db-4e3b-a174-f994660e7ef1 nodeName:}" failed. No retries permitted until 2026-09-17 23:10:59.847347018 +0000 UTC m=+43.213006912 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9572448f-49db-4e3b-a174-f994660e7ef1-metrics-tls") pod "dns-default-qtz2c" (UID: "9572448f-49db-4e3b-a174-f994660e7ef1") : secret "dns-default-metrics-tls" not found
Sep 17 23:10:55.847433 ip-10-0-142-4 kubenswrapper[2584]: E0917 23:10:55.847300    2584 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-75f6fdbdff-5rgvh: secret "image-registry-tls" not found
Sep 17 23:10:55.847433 ip-10-0-142-4 kubenswrapper[2584]: E0917 23:10:55.847420    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/06535038-f186-42c3-b9d1-707e2c7ea285-registry-tls podName:06535038-f186-42c3-b9d1-707e2c7ea285 nodeName:}" failed. No retries permitted until 2026-09-17 23:10:59.847409982 +0000 UTC m=+43.213069876 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/06535038-f186-42c3-b9d1-707e2c7ea285-registry-tls") pod "image-registry-75f6fdbdff-5rgvh" (UID: "06535038-f186-42c3-b9d1-707e2c7ea285") : secret "image-registry-tls" not found
Sep 17 23:10:56.073927 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:56.073739    2584 generic.go:356] "Generic (PLEG): container finished" podID="ea8e5a0a-14f1-4be6-84e1-33567e5d6032" containerID="e2af0d9f71042110736f6fd20b2357f8e25b3a979ee41a7a822a8ffbd986c179" exitCode=0
Sep 17 23:10:56.074350 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:56.073880    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2gjg8" event={"ID":"ea8e5a0a-14f1-4be6-84e1-33567e5d6032","Type":"ContainerDied","Data":"e2af0d9f71042110736f6fd20b2357f8e25b3a979ee41a7a822a8ffbd986c179"}
Sep 17 23:10:57.078802 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:57.078770    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2gjg8" event={"ID":"ea8e5a0a-14f1-4be6-84e1-33567e5d6032","Type":"ContainerStarted","Data":"1d81c7db49ea5e974b4ea0d2e18b94368964c01aac9db1d5cab2585fc1f7f3ec"}
Sep 17 23:10:57.107887 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:57.107829    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-2gjg8" podStartSLOduration=8.206223793 podStartE2EDuration="38.107813397s" totalImagesPullingTime="29.901589604s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 23:10:19 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:10:57.107640251 +0000 UTC m=+40.473300166" watchObservedRunningTime="2026-09-17 23:10:57.107813397 +0000 UTC m=+40.473473312"
Sep 17 23:10:57.236511 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:57.236467    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:10:58.082173 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:58.082132    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-bkxmq" event={"ID":"35051393-90ef-4a48-b95c-a2cc221b538f","Type":"ContainerStarted","Data":"e97ccf4967ac435ddb62bedf6047226218554b8aeaba40787cd09a3d0f4d2e98"}
Sep 17 23:10:58.100720 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:58.100669    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-bkxmq" podStartSLOduration=1.8478145499999998 podStartE2EDuration="6.100649794s" totalImagesPullingTime="4.252835244s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:10:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:10:58.100557071 +0000 UTC m=+41.466216986" watchObservedRunningTime="2026-09-17 23:10:58.100649794 +0000 UTC m=+41.466309712"
Sep 17 23:10:58.762950 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:58.762908    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:10:59.582298 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:59.582269    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:10:59.692255 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:59.692230    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:10:59.876552 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:59.876496    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/06535038-f186-42c3-b9d1-707e2c7ea285-registry-tls\") pod \"image-registry-75f6fdbdff-5rgvh\" (UID: \"06535038-f186-42c3-b9d1-707e2c7ea285\") " pod="openshift-image-registry/image-registry-75f6fdbdff-5rgvh"
Sep 17 23:10:59.876552 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:10:59.876548    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9572448f-49db-4e3b-a174-f994660e7ef1-metrics-tls\") pod \"dns-default-qtz2c\" (UID: \"9572448f-49db-4e3b-a174-f994660e7ef1\") " pod="openshift-dns/dns-default-qtz2c"
Sep 17 23:10:59.876742 ip-10-0-142-4 kubenswrapper[2584]: E0917 23:10:59.876640    2584 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 23:10:59.876742 ip-10-0-142-4 kubenswrapper[2584]: E0917 23:10:59.876653    2584 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-75f6fdbdff-5rgvh: secret "image-registry-tls" not found
Sep 17 23:10:59.876742 ip-10-0-142-4 kubenswrapper[2584]: E0917 23:10:59.876683    2584 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 23:10:59.876742 ip-10-0-142-4 kubenswrapper[2584]: E0917 23:10:59.876694    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/06535038-f186-42c3-b9d1-707e2c7ea285-registry-tls podName:06535038-f186-42c3-b9d1-707e2c7ea285 nodeName:}" failed. No retries permitted until 2026-09-17 23:11:07.876681754 +0000 UTC m=+51.242341648 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/06535038-f186-42c3-b9d1-707e2c7ea285-registry-tls") pod "image-registry-75f6fdbdff-5rgvh" (UID: "06535038-f186-42c3-b9d1-707e2c7ea285") : secret "image-registry-tls" not found
Sep 17 23:10:59.876742 ip-10-0-142-4 kubenswrapper[2584]: E0917 23:10:59.876719    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9572448f-49db-4e3b-a174-f994660e7ef1-metrics-tls podName:9572448f-49db-4e3b-a174-f994660e7ef1 nodeName:}" failed. No retries permitted until 2026-09-17 23:11:07.876708666 +0000 UTC m=+51.242368565 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9572448f-49db-4e3b-a174-f994660e7ef1-metrics-tls") pod "dns-default-qtz2c" (UID: "9572448f-49db-4e3b-a174-f994660e7ef1") : secret "dns-default-metrics-tls" not found
Sep 17 23:11:02.849178 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:11:02.849145    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:11:02.936110 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:11:02.936082    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:11:07.927843 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:11:07.927812    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9572448f-49db-4e3b-a174-f994660e7ef1-metrics-tls\") pod \"dns-default-qtz2c\" (UID: \"9572448f-49db-4e3b-a174-f994660e7ef1\") " pod="openshift-dns/dns-default-qtz2c"
Sep 17 23:11:07.928202 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:11:07.927862    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/06535038-f186-42c3-b9d1-707e2c7ea285-registry-tls\") pod \"image-registry-75f6fdbdff-5rgvh\" (UID: \"06535038-f186-42c3-b9d1-707e2c7ea285\") " pod="openshift-image-registry/image-registry-75f6fdbdff-5rgvh"
Sep 17 23:11:07.928202 ip-10-0-142-4 kubenswrapper[2584]: E0917 23:11:07.927964    2584 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 23:11:07.928202 ip-10-0-142-4 kubenswrapper[2584]: E0917 23:11:07.927966    2584 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 23:11:07.928202 ip-10-0-142-4 kubenswrapper[2584]: E0917 23:11:07.928034    2584 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-75f6fdbdff-5rgvh: secret "image-registry-tls" not found
Sep 17 23:11:07.928202 ip-10-0-142-4 kubenswrapper[2584]: E0917 23:11:07.928042    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9572448f-49db-4e3b-a174-f994660e7ef1-metrics-tls podName:9572448f-49db-4e3b-a174-f994660e7ef1 nodeName:}" failed. No retries permitted until 2026-09-17 23:11:23.928025852 +0000 UTC m=+67.293685759 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9572448f-49db-4e3b-a174-f994660e7ef1-metrics-tls") pod "dns-default-qtz2c" (UID: "9572448f-49db-4e3b-a174-f994660e7ef1") : secret "dns-default-metrics-tls" not found
Sep 17 23:11:07.928202 ip-10-0-142-4 kubenswrapper[2584]: E0917 23:11:07.928071    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/06535038-f186-42c3-b9d1-707e2c7ea285-registry-tls podName:06535038-f186-42c3-b9d1-707e2c7ea285 nodeName:}" failed. No retries permitted until 2026-09-17 23:11:23.928060569 +0000 UTC m=+67.293720463 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/06535038-f186-42c3-b9d1-707e2c7ea285-registry-tls") pod "image-registry-75f6fdbdff-5rgvh" (UID: "06535038-f186-42c3-b9d1-707e2c7ea285") : secret "image-registry-tls" not found
Sep 17 23:11:08.003661 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:11:08.003633    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:11:08.104608 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:11:08.104590    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:11:08.935747 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:11:08.935712    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:11:09.088255 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:11:09.088222    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:11:09.201735 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:11:09.201648    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:11:09.717598 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:11:09.717564    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:11:09.814392 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:11:09.814356    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:11:10.090297 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:11:10.090263    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:11:10.206269 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:11:10.206233    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:11:10.853188 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:11:10.853156    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:11:10.994669 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:11:10.994639    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:11:11.723997 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:11:11.723967    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:11:19.062232 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:11:19.062200    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-tcglq"
Sep 17 23:11:19.256496 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:11:19.256472    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:11:19.350390 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:11:19.350325    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:11:21.410551 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:11:21.410521    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:11:21.494610 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:11:21.494578    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:11:23.940287 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:11:23.940250    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9572448f-49db-4e3b-a174-f994660e7ef1-metrics-tls\") pod \"dns-default-qtz2c\" (UID: \"9572448f-49db-4e3b-a174-f994660e7ef1\") " pod="openshift-dns/dns-default-qtz2c"
Sep 17 23:11:23.940719 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:11:23.940313    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/06535038-f186-42c3-b9d1-707e2c7ea285-registry-tls\") pod \"image-registry-75f6fdbdff-5rgvh\" (UID: \"06535038-f186-42c3-b9d1-707e2c7ea285\") " pod="openshift-image-registry/image-registry-75f6fdbdff-5rgvh"
Sep 17 23:11:23.940719 ip-10-0-142-4 kubenswrapper[2584]: E0917 23:11:23.940377    2584 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 23:11:23.940719 ip-10-0-142-4 kubenswrapper[2584]: E0917 23:11:23.940419    2584 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 23:11:23.940719 ip-10-0-142-4 kubenswrapper[2584]: E0917 23:11:23.940431    2584 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-75f6fdbdff-5rgvh: secret "image-registry-tls" not found
Sep 17 23:11:23.940719 ip-10-0-142-4 kubenswrapper[2584]: E0917 23:11:23.940454    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9572448f-49db-4e3b-a174-f994660e7ef1-metrics-tls podName:9572448f-49db-4e3b-a174-f994660e7ef1 nodeName:}" failed. No retries permitted until 2026-09-17 23:11:55.940436946 +0000 UTC m=+99.306096840 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9572448f-49db-4e3b-a174-f994660e7ef1-metrics-tls") pod "dns-default-qtz2c" (UID: "9572448f-49db-4e3b-a174-f994660e7ef1") : secret "dns-default-metrics-tls" not found
Sep 17 23:11:23.940719 ip-10-0-142-4 kubenswrapper[2584]: E0917 23:11:23.940481    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/06535038-f186-42c3-b9d1-707e2c7ea285-registry-tls podName:06535038-f186-42c3-b9d1-707e2c7ea285 nodeName:}" failed. No retries permitted until 2026-09-17 23:11:55.940470422 +0000 UTC m=+99.306130346 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/06535038-f186-42c3-b9d1-707e2c7ea285-registry-tls") pod "image-registry-75f6fdbdff-5rgvh" (UID: "06535038-f186-42c3-b9d1-707e2c7ea285") : secret "image-registry-tls" not found
Sep 17 23:11:24.443711 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:11:24.443680    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fa08be4f-e950-4dee-a227-feb39c2ba7e0-original-pull-secret\") pod \"global-pull-secret-syncer-24tsm\" (UID: \"fa08be4f-e950-4dee-a227-feb39c2ba7e0\") " pod="kube-system/global-pull-secret-syncer-24tsm"
Sep 17 23:11:24.443854 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:11:24.443728    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d672140f-ad59-4a68-9dde-80b62492af7f-metrics-certs\") pod \"network-metrics-daemon-4bbdp\" (UID: \"d672140f-ad59-4a68-9dde-80b62492af7f\") " pod="openshift-multus/network-metrics-daemon-4bbdp"
Sep 17 23:11:24.447188 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:11:24.447170    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 17 23:11:24.448218 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:11:24.448206    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 17 23:11:24.454333 ip-10-0-142-4 kubenswrapper[2584]: E0917 23:11:24.454319    2584 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 17 23:11:24.454381 ip-10-0-142-4 kubenswrapper[2584]: E0917 23:11:24.454368    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d672140f-ad59-4a68-9dde-80b62492af7f-metrics-certs podName:d672140f-ad59-4a68-9dde-80b62492af7f nodeName:}" failed. No retries permitted until 2026-09-17 23:12:28.454354093 +0000 UTC m=+131.820013988 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d672140f-ad59-4a68-9dde-80b62492af7f-metrics-certs") pod "network-metrics-daemon-4bbdp" (UID: "d672140f-ad59-4a68-9dde-80b62492af7f") : secret "metrics-daemon-secret" not found
Sep 17 23:11:24.457661 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:11:24.457636    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fa08be4f-e950-4dee-a227-feb39c2ba7e0-original-pull-secret\") pod \"global-pull-secret-syncer-24tsm\" (UID: \"fa08be4f-e950-4dee-a227-feb39c2ba7e0\") " pod="kube-system/global-pull-secret-syncer-24tsm"
Sep 17 23:11:24.544354 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:11:24.544333    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bgsh5\" (UniqueName: \"kubernetes.io/projected/a2a26114-4c24-4dce-929c-3cb5caf17e6a-kube-api-access-bgsh5\") pod \"network-check-target-wgps9\" (UID: \"a2a26114-4c24-4dce-929c-3cb5caf17e6a\") " pod="openshift-network-diagnostics/network-check-target-wgps9"
Sep 17 23:11:24.547109 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:11:24.547093    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 17 23:11:24.559444 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:11:24.559426    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 17 23:11:24.566924 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:11:24.566903    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bgsh5\" (UniqueName: \"kubernetes.io/projected/a2a26114-4c24-4dce-929c-3cb5caf17e6a-kube-api-access-bgsh5\") pod \"network-check-target-wgps9\" (UID: \"a2a26114-4c24-4dce-929c-3cb5caf17e6a\") " pod="openshift-network-diagnostics/network-check-target-wgps9"
Sep 17 23:11:24.580058 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:11:24.580045    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-24tsm"
Sep 17 23:11:24.591999 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:11:24.591983    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-jknmn\""
Sep 17 23:11:24.596777 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:11:24.596759    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wgps9"
Sep 17 23:11:24.868062 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:11:24.868037    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-24tsm"]
Sep 17 23:11:24.870956 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:11:24.870931    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa08be4f_e950_4dee_a227_feb39c2ba7e0.slice/crio-e909835d2c83608f3680802c832a9505e059b77b8446d56710c8d6ee67342f8a WatchSource:0}: Error finding container e909835d2c83608f3680802c832a9505e059b77b8446d56710c8d6ee67342f8a: Status 404 returned error can't find the container with id e909835d2c83608f3680802c832a9505e059b77b8446d56710c8d6ee67342f8a
Sep 17 23:11:24.896096 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:11:24.896076    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-wgps9"]
Sep 17 23:11:24.898794 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:11:24.898771    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2a26114_4c24_4dce_929c_3cb5caf17e6a.slice/crio-f13e032062c31f14f490e545b11d3786927d6534632d6914b649fb12b0c1000b WatchSource:0}: Error finding container f13e032062c31f14f490e545b11d3786927d6534632d6914b649fb12b0c1000b: Status 404 returned error can't find the container with id f13e032062c31f14f490e545b11d3786927d6534632d6914b649fb12b0c1000b
Sep 17 23:11:25.143972 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:11:25.143909    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-wgps9" event={"ID":"a2a26114-4c24-4dce-929c-3cb5caf17e6a","Type":"ContainerStarted","Data":"f13e032062c31f14f490e545b11d3786927d6534632d6914b649fb12b0c1000b"}
Sep 17 23:11:25.145029 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:11:25.144997    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-24tsm" event={"ID":"fa08be4f-e950-4dee-a227-feb39c2ba7e0","Type":"ContainerStarted","Data":"e909835d2c83608f3680802c832a9505e059b77b8446d56710c8d6ee67342f8a"}
Sep 17 23:11:25.800642 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:11:25.800611    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:11:25.894227 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:11:25.894198    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:11:26.273672 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:11:26.273632    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:11:26.365420 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:11:26.365383    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:11:27.029997 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:11:27.029959    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:11:27.130184 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:11:27.130153    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:11:30.159248 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:11:30.159214    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-wgps9" event={"ID":"a2a26114-4c24-4dce-929c-3cb5caf17e6a","Type":"ContainerStarted","Data":"35305c64ee385ec06ef4fb7d40ee481236e93e0acb54a7a7e8aff88f5f205699"}
Sep 17 23:11:30.159738 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:11:30.159354    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-wgps9"
Sep 17 23:11:30.160703 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:11:30.160681    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-24tsm" event={"ID":"fa08be4f-e950-4dee-a227-feb39c2ba7e0","Type":"ContainerStarted","Data":"90d0d29df2a06af375ddbfa990c565162a0e613e9096efc1998752369f991447"}
Sep 17 23:11:30.181059 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:11:30.181018    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-wgps9" podStartSLOduration=67.701510095 podStartE2EDuration="1m12.180992433s" totalImagesPullingTime="4.479482338s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:10:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:11:30.179942104 +0000 UTC m=+73.545602019" watchObservedRunningTime="2026-09-17 23:11:30.180992433 +0000 UTC m=+73.546652345"
Sep 17 23:11:30.202883 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:11:30.202841    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-24tsm" podStartSLOduration=66.691508477 podStartE2EDuration="1m11.202829951s" totalImagesPullingTime="4.511321474s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 23:10:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:11:30.201838938 +0000 UTC m=+73.567498854" watchObservedRunningTime="2026-09-17 23:11:30.202829951 +0000 UTC m=+73.568489866"
Sep 17 23:11:30.450374 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:11:30.450297    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:11:30.547017 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:11:30.546985    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:11:34.423320 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:11:34.423288    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:11:34.488190 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:11:34.488159    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:11:34.679246 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:11:34.679185    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:11:34.756935 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:11:34.756912    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:11:35.038346 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:11:35.038318    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-rxcfk"]
Sep 17 23:11:35.040161 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:11:35.040142    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-rxcfk"
Sep 17 23:11:35.040396 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:11:35.040141    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:11:35.044662 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:11:35.044643    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 17 23:11:35.045329 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:11:35.045309    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-fs4cd\""
Sep 17 23:11:35.045448 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:11:35.045311    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 17 23:11:35.045448 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:11:35.045311    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 17 23:11:35.045448 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:11:35.045311    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 17 23:11:35.051841 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:11:35.051821    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-rxcfk"]
Sep 17 23:11:35.116129 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:11:35.116109    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/42b0c2c5-0b9f-4b0c-a782-6df4664999f9-crio-socket\") pod \"insights-runtime-extractor-rxcfk\" (UID: \"42b0c2c5-0b9f-4b0c-a782-6df4664999f9\") " pod="openshift-insights/insights-runtime-extractor-rxcfk"
Sep 17 23:11:35.116232 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:11:35.116134    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/42b0c2c5-0b9f-4b0c-a782-6df4664999f9-data-volume\") pod \"insights-runtime-extractor-rxcfk\" (UID: \"42b0c2c5-0b9f-4b0c-a782-6df4664999f9\") " pod="openshift-insights/insights-runtime-extractor-rxcfk"
Sep 17 23:11:35.116232 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:11:35.116170    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/42b0c2c5-0b9f-4b0c-a782-6df4664999f9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rxcfk\" (UID: \"42b0c2c5-0b9f-4b0c-a782-6df4664999f9\") " pod="openshift-insights/insights-runtime-extractor-rxcfk"
Sep 17 23:11:35.116300 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:11:35.116242    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/42b0c2c5-0b9f-4b0c-a782-6df4664999f9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rxcfk\" (UID: \"42b0c2c5-0b9f-4b0c-a782-6df4664999f9\") " pod="openshift-insights/insights-runtime-extractor-rxcfk"
Sep 17 23:11:35.116336 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:11:35.116301    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bcz4f\" (UniqueName: \"kubernetes.io/projected/42b0c2c5-0b9f-4b0c-a782-6df4664999f9-kube-api-access-bcz4f\") pod \"insights-runtime-extractor-rxcfk\" (UID: \"42b0c2c5-0b9f-4b0c-a782-6df4664999f9\") " pod="openshift-insights/insights-runtime-extractor-rxcfk"
Sep 17 23:11:35.146218 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:11:35.146182    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:11:35.216967 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:11:35.216945    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/42b0c2c5-0b9f-4b0c-a782-6df4664999f9-crio-socket\") pod \"insights-runtime-extractor-rxcfk\" (UID: \"42b0c2c5-0b9f-4b0c-a782-6df4664999f9\") " pod="openshift-insights/insights-runtime-extractor-rxcfk"
Sep 17 23:11:35.217058 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:11:35.216969    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/42b0c2c5-0b9f-4b0c-a782-6df4664999f9-data-volume\") pod \"insights-runtime-extractor-rxcfk\" (UID: \"42b0c2c5-0b9f-4b0c-a782-6df4664999f9\") " pod="openshift-insights/insights-runtime-extractor-rxcfk"
Sep 17 23:11:35.217058 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:11:35.217024    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/42b0c2c5-0b9f-4b0c-a782-6df4664999f9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rxcfk\" (UID: \"42b0c2c5-0b9f-4b0c-a782-6df4664999f9\") " pod="openshift-insights/insights-runtime-extractor-rxcfk"
Sep 17 23:11:35.217124 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:11:35.217078    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/42b0c2c5-0b9f-4b0c-a782-6df4664999f9-crio-socket\") pod \"insights-runtime-extractor-rxcfk\" (UID: \"42b0c2c5-0b9f-4b0c-a782-6df4664999f9\") " pod="openshift-insights/insights-runtime-extractor-rxcfk"
Sep 17 23:11:35.217124 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:11:35.217113    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/42b0c2c5-0b9f-4b0c-a782-6df4664999f9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rxcfk\" (UID: \"42b0c2c5-0b9f-4b0c-a782-6df4664999f9\") " pod="openshift-insights/insights-runtime-extractor-rxcfk"
Sep 17 23:11:35.217188 ip-10-0-142-4 kubenswrapper[2584]: E0917 23:11:35.217128    2584 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 23:11:35.217188 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:11:35.217142    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bcz4f\" (UniqueName: \"kubernetes.io/projected/42b0c2c5-0b9f-4b0c-a782-6df4664999f9-kube-api-access-bcz4f\") pod \"insights-runtime-extractor-rxcfk\" (UID: \"42b0c2c5-0b9f-4b0c-a782-6df4664999f9\") " pod="openshift-insights/insights-runtime-extractor-rxcfk"
Sep 17 23:11:35.217247 ip-10-0-142-4 kubenswrapper[2584]: E0917 23:11:35.217190    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/42b0c2c5-0b9f-4b0c-a782-6df4664999f9-insights-runtime-extractor-tls podName:42b0c2c5-0b9f-4b0c-a782-6df4664999f9 nodeName:}" failed. No retries permitted until 2026-09-17 23:11:35.7171737 +0000 UTC m=+79.082833609 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/42b0c2c5-0b9f-4b0c-a782-6df4664999f9-insights-runtime-extractor-tls") pod "insights-runtime-extractor-rxcfk" (UID: "42b0c2c5-0b9f-4b0c-a782-6df4664999f9") : secret "insights-runtime-extractor-tls" not found
Sep 17 23:11:35.217247 ip-10-0-142-4 kubenswrapper[2584]: E0917 23:11:35.217234    2584 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 23:11:35.217313 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:11:35.217282    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/42b0c2c5-0b9f-4b0c-a782-6df4664999f9-data-volume\") pod \"insights-runtime-extractor-rxcfk\" (UID: \"42b0c2c5-0b9f-4b0c-a782-6df4664999f9\") " pod="openshift-insights/insights-runtime-extractor-rxcfk"
Sep 17 23:11:35.217313 ip-10-0-142-4 kubenswrapper[2584]: E0917 23:11:35.217310    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/42b0c2c5-0b9f-4b0c-a782-6df4664999f9-kube-rbac-proxy-cm podName:42b0c2c5-0b9f-4b0c-a782-6df4664999f9 nodeName:}" failed. No retries permitted until 2026-09-17 23:11:35.717297211 +0000 UTC m=+79.082957124 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/42b0c2c5-0b9f-4b0c-a782-6df4664999f9-kube-rbac-proxy-cm") pod "insights-runtime-extractor-rxcfk" (UID: "42b0c2c5-0b9f-4b0c-a782-6df4664999f9") : configmap "kube-rbac-proxy" not found
Sep 17 23:11:35.228204 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:11:35.228186    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bcz4f\" (UniqueName: \"kubernetes.io/projected/42b0c2c5-0b9f-4b0c-a782-6df4664999f9-kube-api-access-bcz4f\") pod \"insights-runtime-extractor-rxcfk\" (UID: \"42b0c2c5-0b9f-4b0c-a782-6df4664999f9\") " pod="openshift-insights/insights-runtime-extractor-rxcfk"
Sep 17 23:11:35.240347 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:11:35.240316    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:11:35.720994 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:11:35.720961    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/42b0c2c5-0b9f-4b0c-a782-6df4664999f9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rxcfk\" (UID: \"42b0c2c5-0b9f-4b0c-a782-6df4664999f9\") " pod="openshift-insights/insights-runtime-extractor-rxcfk"
Sep 17 23:11:35.720994 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:11:35.720999    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/42b0c2c5-0b9f-4b0c-a782-6df4664999f9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rxcfk\" (UID: \"42b0c2c5-0b9f-4b0c-a782-6df4664999f9\") " pod="openshift-insights/insights-runtime-extractor-rxcfk"
Sep 17 23:11:35.721351 ip-10-0-142-4 kubenswrapper[2584]: E0917 23:11:35.721075    2584 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 23:11:35.721351 ip-10-0-142-4 kubenswrapper[2584]: E0917 23:11:35.721113    2584 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 23:11:35.721351 ip-10-0-142-4 kubenswrapper[2584]: E0917 23:11:35.721125    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/42b0c2c5-0b9f-4b0c-a782-6df4664999f9-kube-rbac-proxy-cm podName:42b0c2c5-0b9f-4b0c-a782-6df4664999f9 nodeName:}" failed. No retries permitted until 2026-09-17 23:11:36.721112115 +0000 UTC m=+80.086772009 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/42b0c2c5-0b9f-4b0c-a782-6df4664999f9-kube-rbac-proxy-cm") pod "insights-runtime-extractor-rxcfk" (UID: "42b0c2c5-0b9f-4b0c-a782-6df4664999f9") : configmap "kube-rbac-proxy" not found
Sep 17 23:11:35.721351 ip-10-0-142-4 kubenswrapper[2584]: E0917 23:11:35.721194    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/42b0c2c5-0b9f-4b0c-a782-6df4664999f9-insights-runtime-extractor-tls podName:42b0c2c5-0b9f-4b0c-a782-6df4664999f9 nodeName:}" failed. No retries permitted until 2026-09-17 23:11:36.721177691 +0000 UTC m=+80.086837585 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/42b0c2c5-0b9f-4b0c-a782-6df4664999f9-insights-runtime-extractor-tls") pod "insights-runtime-extractor-rxcfk" (UID: "42b0c2c5-0b9f-4b0c-a782-6df4664999f9") : secret "insights-runtime-extractor-tls" not found
Sep 17 23:11:36.727731 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:11:36.727699    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/42b0c2c5-0b9f-4b0c-a782-6df4664999f9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rxcfk\" (UID: \"42b0c2c5-0b9f-4b0c-a782-6df4664999f9\") " pod="openshift-insights/insights-runtime-extractor-rxcfk"
Sep 17 23:11:36.727731 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:11:36.727734    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/42b0c2c5-0b9f-4b0c-a782-6df4664999f9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rxcfk\" (UID: \"42b0c2c5-0b9f-4b0c-a782-6df4664999f9\") " pod="openshift-insights/insights-runtime-extractor-rxcfk"
Sep 17 23:11:36.728143 ip-10-0-142-4 kubenswrapper[2584]: E0917 23:11:36.727832    2584 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 23:11:36.728143 ip-10-0-142-4 kubenswrapper[2584]: E0917 23:11:36.727842    2584 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 23:11:36.728143 ip-10-0-142-4 kubenswrapper[2584]: E0917 23:11:36.727892    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/42b0c2c5-0b9f-4b0c-a782-6df4664999f9-insights-runtime-extractor-tls podName:42b0c2c5-0b9f-4b0c-a782-6df4664999f9 nodeName:}" failed. No retries permitted until 2026-09-17 23:11:38.727875878 +0000 UTC m=+82.093535786 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/42b0c2c5-0b9f-4b0c-a782-6df4664999f9-insights-runtime-extractor-tls") pod "insights-runtime-extractor-rxcfk" (UID: "42b0c2c5-0b9f-4b0c-a782-6df4664999f9") : secret "insights-runtime-extractor-tls" not found
Sep 17 23:11:36.728143 ip-10-0-142-4 kubenswrapper[2584]: E0917 23:11:36.727906    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/42b0c2c5-0b9f-4b0c-a782-6df4664999f9-kube-rbac-proxy-cm podName:42b0c2c5-0b9f-4b0c-a782-6df4664999f9 nodeName:}" failed. No retries permitted until 2026-09-17 23:11:38.727899479 +0000 UTC m=+82.093559373 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/42b0c2c5-0b9f-4b0c-a782-6df4664999f9-kube-rbac-proxy-cm") pod "insights-runtime-extractor-rxcfk" (UID: "42b0c2c5-0b9f-4b0c-a782-6df4664999f9") : configmap "kube-rbac-proxy" not found
Sep 17 23:11:36.785885 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:11:36.785864    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-895t6_fcc7ae4e-ace4-490f-a4e0-1c7c0ea22f2d/dns-node-resolver/0.log"
Sep 17 23:11:36.859274 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:11:36.859250    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:11:36.962948 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:11:36.962911    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:11:37.583324 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:11:37.583294    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-5spz7_04f96656-0988-4bea-a70f-79d1b8445201/node-ca/0.log"
Sep 17 23:11:37.908035 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:11:37.907933    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:11:38.584321 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:11:38.584295    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-bkxmq_35051393-90ef-4a48-b95c-a2cc221b538f/serve-healthcheck-canary/0.log"
Sep 17 23:11:38.742264 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:11:38.742237    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/42b0c2c5-0b9f-4b0c-a782-6df4664999f9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rxcfk\" (UID: \"42b0c2c5-0b9f-4b0c-a782-6df4664999f9\") " pod="openshift-insights/insights-runtime-extractor-rxcfk"
Sep 17 23:11:38.742405 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:11:38.742270    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/42b0c2c5-0b9f-4b0c-a782-6df4664999f9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rxcfk\" (UID: \"42b0c2c5-0b9f-4b0c-a782-6df4664999f9\") " pod="openshift-insights/insights-runtime-extractor-rxcfk"
Sep 17 23:11:38.742405 ip-10-0-142-4 kubenswrapper[2584]: E0917 23:11:38.742328    2584 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 23:11:38.742405 ip-10-0-142-4 kubenswrapper[2584]: E0917 23:11:38.742387    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/42b0c2c5-0b9f-4b0c-a782-6df4664999f9-kube-rbac-proxy-cm podName:42b0c2c5-0b9f-4b0c-a782-6df4664999f9 nodeName:}" failed. No retries permitted until 2026-09-17 23:11:42.742373088 +0000 UTC m=+86.108032995 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/42b0c2c5-0b9f-4b0c-a782-6df4664999f9-kube-rbac-proxy-cm") pod "insights-runtime-extractor-rxcfk" (UID: "42b0c2c5-0b9f-4b0c-a782-6df4664999f9") : configmap "kube-rbac-proxy" not found
Sep 17 23:11:38.742561 ip-10-0-142-4 kubenswrapper[2584]: E0917 23:11:38.742444    2584 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 23:11:38.742561 ip-10-0-142-4 kubenswrapper[2584]: E0917 23:11:38.742514    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/42b0c2c5-0b9f-4b0c-a782-6df4664999f9-insights-runtime-extractor-tls podName:42b0c2c5-0b9f-4b0c-a782-6df4664999f9 nodeName:}" failed. No retries permitted until 2026-09-17 23:11:42.742497682 +0000 UTC m=+86.108157580 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/42b0c2c5-0b9f-4b0c-a782-6df4664999f9-insights-runtime-extractor-tls") pod "insights-runtime-extractor-rxcfk" (UID: "42b0c2c5-0b9f-4b0c-a782-6df4664999f9") : secret "insights-runtime-extractor-tls" not found
Sep 17 23:11:41.495852 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:11:41.495819    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:11:41.694761 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:11:41.694732    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:11:41.888511 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:11:41.888483    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:11:42.591165 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:11:42.591142    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:11:42.769174 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:11:42.769145    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/42b0c2c5-0b9f-4b0c-a782-6df4664999f9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rxcfk\" (UID: \"42b0c2c5-0b9f-4b0c-a782-6df4664999f9\") " pod="openshift-insights/insights-runtime-extractor-rxcfk"
Sep 17 23:11:42.769318 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:11:42.769183    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/42b0c2c5-0b9f-4b0c-a782-6df4664999f9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rxcfk\" (UID: \"42b0c2c5-0b9f-4b0c-a782-6df4664999f9\") " pod="openshift-insights/insights-runtime-extractor-rxcfk"
Sep 17 23:11:42.769318 ip-10-0-142-4 kubenswrapper[2584]: E0917 23:11:42.769260    2584 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 23:11:42.769318 ip-10-0-142-4 kubenswrapper[2584]: E0917 23:11:42.769283    2584 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 23:11:42.769318 ip-10-0-142-4 kubenswrapper[2584]: E0917 23:11:42.769306    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/42b0c2c5-0b9f-4b0c-a782-6df4664999f9-kube-rbac-proxy-cm podName:42b0c2c5-0b9f-4b0c-a782-6df4664999f9 nodeName:}" failed. No retries permitted until 2026-09-17 23:11:50.769294434 +0000 UTC m=+94.134954328 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/42b0c2c5-0b9f-4b0c-a782-6df4664999f9-kube-rbac-proxy-cm") pod "insights-runtime-extractor-rxcfk" (UID: "42b0c2c5-0b9f-4b0c-a782-6df4664999f9") : configmap "kube-rbac-proxy" not found
Sep 17 23:11:42.769472 ip-10-0-142-4 kubenswrapper[2584]: E0917 23:11:42.769335    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/42b0c2c5-0b9f-4b0c-a782-6df4664999f9-insights-runtime-extractor-tls podName:42b0c2c5-0b9f-4b0c-a782-6df4664999f9 nodeName:}" failed. No retries permitted until 2026-09-17 23:11:50.769323573 +0000 UTC m=+94.134983468 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/42b0c2c5-0b9f-4b0c-a782-6df4664999f9-insights-runtime-extractor-tls") pod "insights-runtime-extractor-rxcfk" (UID: "42b0c2c5-0b9f-4b0c-a782-6df4664999f9") : secret "insights-runtime-extractor-tls" not found
Sep 17 23:11:43.072933 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:11:43.072901    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:11:43.313935 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:11:43.313885    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:11:43.752110 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:11:43.752083    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:11:43.986218 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:11:43.986194    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:11:44.337015 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:11:44.336984    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:11:44.562793 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:11:44.562761    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:11:44.914709 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:11:44.914670    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:11:45.101093 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:11:45.101063    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:11:45.294398 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:11:45.294368    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:11:45.504265 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:11:45.504231    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:11:45.641634 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:11:45.641561    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:11:45.795171 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:11:45.795142    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:11:46.686884 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:11:46.686849    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:11:47.098300 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:11:47.098271    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:11:47.302447 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:11:47.302417    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:11:47.418199 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:11:47.418126    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:11:47.558088 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:11:47.558062    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:11:47.728146 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:11:47.728073    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:11:47.875671 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:11:47.875645    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:11:48.684777 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:11:48.684726    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:11:49.076527 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:11:49.076498    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:11:49.285292 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:11:49.285263    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:11:49.515553 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:11:49.515528    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:11:49.625804 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:11:49.625753    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:11:49.905283 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:11:49.905201    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:11:50.487025 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:11:50.486971    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:11:50.829082 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:11:50.829048    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/42b0c2c5-0b9f-4b0c-a782-6df4664999f9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rxcfk\" (UID: \"42b0c2c5-0b9f-4b0c-a782-6df4664999f9\") " pod="openshift-insights/insights-runtime-extractor-rxcfk"
Sep 17 23:11:50.829250 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:11:50.829089    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/42b0c2c5-0b9f-4b0c-a782-6df4664999f9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rxcfk\" (UID: \"42b0c2c5-0b9f-4b0c-a782-6df4664999f9\") " pod="openshift-insights/insights-runtime-extractor-rxcfk"
Sep 17 23:11:50.829250 ip-10-0-142-4 kubenswrapper[2584]: E0917 23:11:50.829198    2584 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 23:11:50.829369 ip-10-0-142-4 kubenswrapper[2584]: E0917 23:11:50.829279    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/42b0c2c5-0b9f-4b0c-a782-6df4664999f9-insights-runtime-extractor-tls podName:42b0c2c5-0b9f-4b0c-a782-6df4664999f9 nodeName:}" failed. No retries permitted until 2026-09-17 23:12:06.829258679 +0000 UTC m=+110.194918590 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/42b0c2c5-0b9f-4b0c-a782-6df4664999f9-insights-runtime-extractor-tls") pod "insights-runtime-extractor-rxcfk" (UID: "42b0c2c5-0b9f-4b0c-a782-6df4664999f9") : secret "insights-runtime-extractor-tls" not found
Sep 17 23:11:50.829567 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:11:50.829545    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/42b0c2c5-0b9f-4b0c-a782-6df4664999f9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rxcfk\" (UID: \"42b0c2c5-0b9f-4b0c-a782-6df4664999f9\") " pod="openshift-insights/insights-runtime-extractor-rxcfk"
Sep 17 23:11:51.128151 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:11:51.128046    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:11:51.342687 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:11:51.342658    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:11:51.703998 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:11:51.703974    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:11:51.948390 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:11:51.948353    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:11:52.087099 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:11:52.087069    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:11:55.965121 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:11:55.965081    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/06535038-f186-42c3-b9d1-707e2c7ea285-registry-tls\") pod \"image-registry-75f6fdbdff-5rgvh\" (UID: \"06535038-f186-42c3-b9d1-707e2c7ea285\") " pod="openshift-image-registry/image-registry-75f6fdbdff-5rgvh"
Sep 17 23:11:55.965546 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:11:55.965148    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9572448f-49db-4e3b-a174-f994660e7ef1-metrics-tls\") pod \"dns-default-qtz2c\" (UID: \"9572448f-49db-4e3b-a174-f994660e7ef1\") " pod="openshift-dns/dns-default-qtz2c"
Sep 17 23:11:55.967492 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:11:55.967471    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9572448f-49db-4e3b-a174-f994660e7ef1-metrics-tls\") pod \"dns-default-qtz2c\" (UID: \"9572448f-49db-4e3b-a174-f994660e7ef1\") " pod="openshift-dns/dns-default-qtz2c"
Sep 17 23:11:55.967645 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:11:55.967628    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/06535038-f186-42c3-b9d1-707e2c7ea285-registry-tls\") pod \"image-registry-75f6fdbdff-5rgvh\" (UID: \"06535038-f186-42c3-b9d1-707e2c7ea285\") " pod="openshift-image-registry/image-registry-75f6fdbdff-5rgvh"
Sep 17 23:11:55.999190 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:11:55.999167    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-5987r\""
Sep 17 23:11:56.007099 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:11:56.007083    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-75f6fdbdff-5rgvh"
Sep 17 23:11:56.036123 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:11:56.036101    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-42llv\""
Sep 17 23:11:56.044322 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:11:56.044299    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-qtz2c"
Sep 17 23:11:56.288554 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:11:56.288526    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-75f6fdbdff-5rgvh"]
Sep 17 23:11:56.291117 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:11:56.291093    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06535038_f186_42c3_b9d1_707e2c7ea285.slice/crio-9c412698d32fe9f3c7257914d99b86c225846f63d9f212d26bd475c962971aa9 WatchSource:0}: Error finding container 9c412698d32fe9f3c7257914d99b86c225846f63d9f212d26bd475c962971aa9: Status 404 returned error can't find the container with id 9c412698d32fe9f3c7257914d99b86c225846f63d9f212d26bd475c962971aa9
Sep 17 23:11:56.317468 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:11:56.317449    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-qtz2c"]
Sep 17 23:11:56.321901 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:11:56.321878    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9572448f_49db_4e3b_a174_f994660e7ef1.slice/crio-e5622e59d2d60779484ad75067453f9e37db5bfb4f4d67bf9d276aa0d13ce1b9 WatchSource:0}: Error finding container e5622e59d2d60779484ad75067453f9e37db5bfb4f4d67bf9d276aa0d13ce1b9: Status 404 returned error can't find the container with id e5622e59d2d60779484ad75067453f9e37db5bfb4f4d67bf9d276aa0d13ce1b9
Sep 17 23:11:57.223167 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:11:57.223067    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-75f6fdbdff-5rgvh" event={"ID":"06535038-f186-42c3-b9d1-707e2c7ea285","Type":"ContainerStarted","Data":"f1f191b3b0e62d08218cb5c0b74f2abd552d690950d7a5e0c9010f83ed0d22e0"}
Sep 17 23:11:57.223167 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:11:57.223127    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-75f6fdbdff-5rgvh" event={"ID":"06535038-f186-42c3-b9d1-707e2c7ea285","Type":"ContainerStarted","Data":"9c412698d32fe9f3c7257914d99b86c225846f63d9f212d26bd475c962971aa9"}
Sep 17 23:11:57.223626 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:11:57.223184    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-75f6fdbdff-5rgvh"
Sep 17 23:11:57.224307 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:11:57.224281    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-qtz2c" event={"ID":"9572448f-49db-4e3b-a174-f994660e7ef1","Type":"ContainerStarted","Data":"e5622e59d2d60779484ad75067453f9e37db5bfb4f4d67bf9d276aa0d13ce1b9"}
Sep 17 23:11:57.257573 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:11:57.257450    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-75f6fdbdff-5rgvh" podStartSLOduration=98.25743496 podStartE2EDuration="1m38.25743496s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:10:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:11:57.255950601 +0000 UTC m=+100.621610519" watchObservedRunningTime="2026-09-17 23:11:57.25743496 +0000 UTC m=+100.623094875"
Sep 17 23:11:58.229167 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:11:58.229089    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-qtz2c" event={"ID":"9572448f-49db-4e3b-a174-f994660e7ef1","Type":"ContainerStarted","Data":"a173cbb84c60df4c01b09cf18f2f3f2877511cb33ab1c94743599218f4e18797"}
Sep 17 23:11:58.229167 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:11:58.229137    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-qtz2c" event={"ID":"9572448f-49db-4e3b-a174-f994660e7ef1","Type":"ContainerStarted","Data":"8a3ca119a00a877b095c15cb2b751f14f36cfbc5a1e4f7653fa8dbc35c245995"}
Sep 17 23:11:58.249945 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:11:58.249901    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-qtz2c" podStartSLOduration=64.7317737 podStartE2EDuration="1m6.249886735s" totalImagesPullingTime="1.518113035s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:10:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:11:58.248809644 +0000 UTC m=+101.614469562" watchObservedRunningTime="2026-09-17 23:11:58.249886735 +0000 UTC m=+101.615546651"
Sep 17 23:11:59.232881 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:11:59.232840    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-qtz2c"
Sep 17 23:11:59.577695 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:11:59.577664    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:11:59.663909 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:11:59.663881    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:12:01.165879 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:01.165850    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-wgps9"
Sep 17 23:12:01.695161 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:01.695127    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-bcd668cbd-vj764"]
Sep 17 23:12:01.721793 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:01.721765    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-bcd668cbd-vj764"
Sep 17 23:12:01.724147 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:01.724123    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-bcd668cbd-vj764"]
Sep 17 23:12:01.724459 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:01.724439    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 17 23:12:01.725682 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:01.725663    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Sep 17 23:12:01.725784 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:01.725687    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 17 23:12:01.725784 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:01.725725    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 17 23:12:01.785678 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:01.785648    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-7cbc45fddc-fd98x"]
Sep 17 23:12:01.806995 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:01.806974    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5a954d16-8519-4492-bab9-245f009e89ac-tmp\") pod \"klusterlet-addon-workmgr-bcd668cbd-vj764\" (UID: \"5a954d16-8519-4492-bab9-245f009e89ac\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-bcd668cbd-vj764"
Sep 17 23:12:01.807095 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:01.807024    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/5a954d16-8519-4492-bab9-245f009e89ac-klusterlet-config\") pod \"klusterlet-addon-workmgr-bcd668cbd-vj764\" (UID: \"5a954d16-8519-4492-bab9-245f009e89ac\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-bcd668cbd-vj764"
Sep 17 23:12:01.807134 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:01.807094    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fs7cp\" (UniqueName: \"kubernetes.io/projected/5a954d16-8519-4492-bab9-245f009e89ac-kube-api-access-fs7cp\") pod \"klusterlet-addon-workmgr-bcd668cbd-vj764\" (UID: \"5a954d16-8519-4492-bab9-245f009e89ac\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-bcd668cbd-vj764"
Sep 17 23:12:01.817387 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:01.817372    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7cbc45fddc-fd98x"]
Sep 17 23:12:01.817456 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:01.817449    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-fd98x"
Sep 17 23:12:01.836328 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:01.836307    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-7sh24\""
Sep 17 23:12:01.908167 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:01.908137    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5a954d16-8519-4492-bab9-245f009e89ac-tmp\") pod \"klusterlet-addon-workmgr-bcd668cbd-vj764\" (UID: \"5a954d16-8519-4492-bab9-245f009e89ac\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-bcd668cbd-vj764"
Sep 17 23:12:01.908167 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:01.908169    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/5a954d16-8519-4492-bab9-245f009e89ac-klusterlet-config\") pod \"klusterlet-addon-workmgr-bcd668cbd-vj764\" (UID: \"5a954d16-8519-4492-bab9-245f009e89ac\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-bcd668cbd-vj764"
Sep 17 23:12:01.908359 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:01.908193    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m9t6k\" (UniqueName: \"kubernetes.io/projected/8e6fd4b4-820c-49b2-81f1-45d3587a05be-kube-api-access-m9t6k\") pod \"network-check-source-7cbc45fddc-fd98x\" (UID: \"8e6fd4b4-820c-49b2-81f1-45d3587a05be\") " pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-fd98x"
Sep 17 23:12:01.908359 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:01.908246    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fs7cp\" (UniqueName: \"kubernetes.io/projected/5a954d16-8519-4492-bab9-245f009e89ac-kube-api-access-fs7cp\") pod \"klusterlet-addon-workmgr-bcd668cbd-vj764\" (UID: \"5a954d16-8519-4492-bab9-245f009e89ac\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-bcd668cbd-vj764"
Sep 17 23:12:01.908566 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:01.908548    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5a954d16-8519-4492-bab9-245f009e89ac-tmp\") pod \"klusterlet-addon-workmgr-bcd668cbd-vj764\" (UID: \"5a954d16-8519-4492-bab9-245f009e89ac\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-bcd668cbd-vj764"
Sep 17 23:12:01.910718 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:01.910693    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/5a954d16-8519-4492-bab9-245f009e89ac-klusterlet-config\") pod \"klusterlet-addon-workmgr-bcd668cbd-vj764\" (UID: \"5a954d16-8519-4492-bab9-245f009e89ac\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-bcd668cbd-vj764"
Sep 17 23:12:02.009520 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:02.009485    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m9t6k\" (UniqueName: \"kubernetes.io/projected/8e6fd4b4-820c-49b2-81f1-45d3587a05be-kube-api-access-m9t6k\") pod \"network-check-source-7cbc45fddc-fd98x\" (UID: \"8e6fd4b4-820c-49b2-81f1-45d3587a05be\") " pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-fd98x"
Sep 17 23:12:02.029409 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:02.029388    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fs7cp\" (UniqueName: \"kubernetes.io/projected/5a954d16-8519-4492-bab9-245f009e89ac-kube-api-access-fs7cp\") pod \"klusterlet-addon-workmgr-bcd668cbd-vj764\" (UID: \"5a954d16-8519-4492-bab9-245f009e89ac\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-bcd668cbd-vj764"
Sep 17 23:12:02.033170 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:02.033153    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-bcd668cbd-vj764"
Sep 17 23:12:02.053351 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:02.053323    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:12:02.060386 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:02.060369    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m9t6k\" (UniqueName: \"kubernetes.io/projected/8e6fd4b4-820c-49b2-81f1-45d3587a05be-kube-api-access-m9t6k\") pod \"network-check-source-7cbc45fddc-fd98x\" (UID: \"8e6fd4b4-820c-49b2-81f1-45d3587a05be\") " pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-fd98x"
Sep 17 23:12:02.124938 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:02.124905    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-fd98x"
Sep 17 23:12:02.293237 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:02.293155    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:12:02.348202 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:02.347367    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-bcd668cbd-vj764"]
Sep 17 23:12:02.349900 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:12:02.349878    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a954d16_8519_4492_bab9_245f009e89ac.slice/crio-dbda9af0a10cb18dc89e536b591dccde917393e6b05954e7c88e27cc524b2fc6 WatchSource:0}: Error finding container dbda9af0a10cb18dc89e536b591dccde917393e6b05954e7c88e27cc524b2fc6: Status 404 returned error can't find the container with id dbda9af0a10cb18dc89e536b591dccde917393e6b05954e7c88e27cc524b2fc6
Sep 17 23:12:02.420221 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:02.420198    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7cbc45fddc-fd98x"]
Sep 17 23:12:02.422851 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:12:02.422824    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e6fd4b4_820c_49b2_81f1_45d3587a05be.slice/crio-1fc03bd7fd3b0c62f30ae5f1f3cc26bca9da31682f5a6d7a7eb5d1bcb29d7e9d WatchSource:0}: Error finding container 1fc03bd7fd3b0c62f30ae5f1f3cc26bca9da31682f5a6d7a7eb5d1bcb29d7e9d: Status 404 returned error can't find the container with id 1fc03bd7fd3b0c62f30ae5f1f3cc26bca9da31682f5a6d7a7eb5d1bcb29d7e9d
Sep 17 23:12:02.471615 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:02.471596    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:12:03.256947 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:03.256660    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-fd98x" event={"ID":"8e6fd4b4-820c-49b2-81f1-45d3587a05be","Type":"ContainerStarted","Data":"0b1e6980c40f065c9e6872af0440eab691abe68e29c4d1afd3a992d21c1b689e"}
Sep 17 23:12:03.256947 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:03.256743    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-fd98x" event={"ID":"8e6fd4b4-820c-49b2-81f1-45d3587a05be","Type":"ContainerStarted","Data":"1fc03bd7fd3b0c62f30ae5f1f3cc26bca9da31682f5a6d7a7eb5d1bcb29d7e9d"}
Sep 17 23:12:03.258931 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:03.258898    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-bcd668cbd-vj764" event={"ID":"5a954d16-8519-4492-bab9-245f009e89ac","Type":"ContainerStarted","Data":"dbda9af0a10cb18dc89e536b591dccde917393e6b05954e7c88e27cc524b2fc6"}
Sep 17 23:12:03.278733 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:03.278326    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-fd98x" podStartSLOduration=2.2783060920000002 podStartE2EDuration="2.278306092s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:12:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:12:03.276934107 +0000 UTC m=+106.642646448" watchObservedRunningTime="2026-09-17 23:12:03.278306092 +0000 UTC m=+106.643966009"
Sep 17 23:12:05.028735 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:05.028695    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-6594f4c75-c9vsj"]
Sep 17 23:12:05.049919 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:05.049889    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6594f4c75-c9vsj"]
Sep 17 23:12:05.050109 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:05.050000    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6594f4c75-c9vsj"
Sep 17 23:12:05.051225 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:05.051198    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:12:05.053632 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:05.053365    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 17 23:12:05.053632 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:05.053433    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-rdx89\""
Sep 17 23:12:05.053632 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:05.053472    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 17 23:12:05.148274 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:05.148231    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7xxfq\" (UniqueName: \"kubernetes.io/projected/72436ae5-af59-40fa-8525-c27cf3209e34-kube-api-access-7xxfq\") pod \"downloads-6594f4c75-c9vsj\" (UID: \"72436ae5-af59-40fa-8525-c27cf3209e34\") " pod="openshift-console/downloads-6594f4c75-c9vsj"
Sep 17 23:12:05.169424 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:05.169394    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:12:05.249435 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:05.249387    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7xxfq\" (UniqueName: \"kubernetes.io/projected/72436ae5-af59-40fa-8525-c27cf3209e34-kube-api-access-7xxfq\") pod \"downloads-6594f4c75-c9vsj\" (UID: \"72436ae5-af59-40fa-8525-c27cf3209e34\") " pod="openshift-console/downloads-6594f4c75-c9vsj"
Sep 17 23:12:05.258379 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:05.258358    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7xxfq\" (UniqueName: \"kubernetes.io/projected/72436ae5-af59-40fa-8525-c27cf3209e34-kube-api-access-7xxfq\") pod \"downloads-6594f4c75-c9vsj\" (UID: \"72436ae5-af59-40fa-8525-c27cf3209e34\") " pod="openshift-console/downloads-6594f4c75-c9vsj"
Sep 17 23:12:05.311112 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:05.311020    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:12:05.361136 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:05.361107    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6594f4c75-c9vsj"
Sep 17 23:12:05.438712 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:05.438677    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:12:05.523400 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:05.523363    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:12:05.623110 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:05.622993    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:12:05.743876 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:05.743835    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:12:06.167066 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:06.166986    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6594f4c75-c9vsj"]
Sep 17 23:12:06.169710 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:12:06.169682    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72436ae5_af59_40fa_8525_c27cf3209e34.slice/crio-120e6be02888a8ed28226960f8bcc8793f0ffad3a8d3eae97515603c4b5fae48 WatchSource:0}: Error finding container 120e6be02888a8ed28226960f8bcc8793f0ffad3a8d3eae97515603c4b5fae48: Status 404 returned error can't find the container with id 120e6be02888a8ed28226960f8bcc8793f0ffad3a8d3eae97515603c4b5fae48
Sep 17 23:12:06.270154 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:06.270126    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-bcd668cbd-vj764" event={"ID":"5a954d16-8519-4492-bab9-245f009e89ac","Type":"ContainerStarted","Data":"23d9ea7f4e118764532dfd4cb771c54a8ebeea5e9781e5e4df8f287fa1cb267b"}
Sep 17 23:12:06.270337 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:06.270318    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-bcd668cbd-vj764"
Sep 17 23:12:06.271368 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:06.271343    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6594f4c75-c9vsj" event={"ID":"72436ae5-af59-40fa-8525-c27cf3209e34","Type":"ContainerStarted","Data":"120e6be02888a8ed28226960f8bcc8793f0ffad3a8d3eae97515603c4b5fae48"}
Sep 17 23:12:06.272052 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:06.272033    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-bcd668cbd-vj764"
Sep 17 23:12:06.290526 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:06.290487    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-bcd668cbd-vj764" podStartSLOduration=1.7156045180000001 podStartE2EDuration="5.2904704s" totalImagesPullingTime="3.574865882s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:12:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:12:06.290122891 +0000 UTC m=+109.655782807" watchObservedRunningTime="2026-09-17 23:12:06.2904704 +0000 UTC m=+109.656130321"
Sep 17 23:12:06.540623 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:06.540590    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:12:06.642350 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:06.642322    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:12:06.735390 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:06.735363    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:12:06.850860 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:06.850783    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:12:06.862903 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:06.862823    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/42b0c2c5-0b9f-4b0c-a782-6df4664999f9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rxcfk\" (UID: \"42b0c2c5-0b9f-4b0c-a782-6df4664999f9\") " pod="openshift-insights/insights-runtime-extractor-rxcfk"
Sep 17 23:12:06.865657 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:06.865634    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/42b0c2c5-0b9f-4b0c-a782-6df4664999f9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rxcfk\" (UID: \"42b0c2c5-0b9f-4b0c-a782-6df4664999f9\") " pod="openshift-insights/insights-runtime-extractor-rxcfk"
Sep 17 23:12:06.960455 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:06.960424    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:12:07.148669 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:07.148587    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-rxcfk"
Sep 17 23:12:07.410414 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:07.410331    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:12:07.432435 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:07.432409    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-rxcfk"]
Sep 17 23:12:07.436251 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:12:07.436218    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42b0c2c5_0b9f_4b0c_a782_6df4664999f9.slice/crio-ef1206d9e211ad0c1aeea09419a5cead2e765447178138c960a8d58d353d6763 WatchSource:0}: Error finding container ef1206d9e211ad0c1aeea09419a5cead2e765447178138c960a8d58d353d6763: Status 404 returned error can't find the container with id ef1206d9e211ad0c1aeea09419a5cead2e765447178138c960a8d58d353d6763
Sep 17 23:12:08.286826 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:08.286790    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-rxcfk" event={"ID":"42b0c2c5-0b9f-4b0c-a782-6df4664999f9","Type":"ContainerStarted","Data":"83ae915b5d8155f46e96b8fccf2130b5492a41cf04dfae4e8c6319e7d8e63cbe"}
Sep 17 23:12:08.287130 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:08.287107    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-rxcfk" event={"ID":"42b0c2c5-0b9f-4b0c-a782-6df4664999f9","Type":"ContainerStarted","Data":"ef1206d9e211ad0c1aeea09419a5cead2e765447178138c960a8d58d353d6763"}
Sep 17 23:12:08.675964 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:08.675850    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:12:08.921558 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:08.921523    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:12:09.232232 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:09.232181    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:12:09.237780 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:09.237755    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-qtz2c"
Sep 17 23:12:09.293496 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:09.293460    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-rxcfk" event={"ID":"42b0c2c5-0b9f-4b0c-a782-6df4664999f9","Type":"ContainerStarted","Data":"006cf5e49a07b05cc76e479917eb7ae546b17df7fadda7fb7a1a32b457bcdcb5"}
Sep 17 23:12:09.375145 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:09.375116    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:12:09.549132 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:09.549097    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:12:09.733666 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:09.733637    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:12:09.842349 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:09.842275    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-76794cbd6c-zd4nf"]
Sep 17 23:12:09.844711 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:09.844688    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-zd4nf"
Sep 17 23:12:09.853162 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:09.853142    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Sep 17 23:12:09.853887 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:09.853861    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Sep 17 23:12:09.853996 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:09.853868    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-ckvp8\""
Sep 17 23:12:09.866303 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:09.866280    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-76794cbd6c-zd4nf"]
Sep 17 23:12:09.937321 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:09.937284    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:12:09.986899 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:09.986863    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kh4cm\" (UniqueName: \"kubernetes.io/projected/88bdd5cd-67c8-4000-8dca-5c5875512672-kube-api-access-kh4cm\") pod \"migrator-76794cbd6c-zd4nf\" (UID: \"88bdd5cd-67c8-4000-8dca-5c5875512672\") " pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-zd4nf"
Sep 17 23:12:10.069228 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:10.069190    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:12:10.088296 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:10.088267    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kh4cm\" (UniqueName: \"kubernetes.io/projected/88bdd5cd-67c8-4000-8dca-5c5875512672-kube-api-access-kh4cm\") pod \"migrator-76794cbd6c-zd4nf\" (UID: \"88bdd5cd-67c8-4000-8dca-5c5875512672\") " pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-zd4nf"
Sep 17 23:12:10.111581 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:10.111524    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kh4cm\" (UniqueName: \"kubernetes.io/projected/88bdd5cd-67c8-4000-8dca-5c5875512672-kube-api-access-kh4cm\") pod \"migrator-76794cbd6c-zd4nf\" (UID: \"88bdd5cd-67c8-4000-8dca-5c5875512672\") " pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-zd4nf"
Sep 17 23:12:10.156377 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:10.156345    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-zd4nf"
Sep 17 23:12:10.299972 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:10.299943    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-rxcfk" event={"ID":"42b0c2c5-0b9f-4b0c-a782-6df4664999f9","Type":"ContainerStarted","Data":"c260dae1e27c7f7c6d518bcea0ed847653ac3b0f15364bf9aa404e3f8809897d"}
Sep 17 23:12:10.339267 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:10.339226    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-rxcfk" podStartSLOduration=32.711800679 podStartE2EDuration="35.339208865s" totalImagesPullingTime="2.627408186s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 23:11:35 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:12:10.337862622 +0000 UTC m=+113.703522540" watchObservedRunningTime="2026-09-17 23:12:10.339208865 +0000 UTC m=+113.704868787"
Sep 17 23:12:10.448060 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:10.447972    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-76794cbd6c-zd4nf"]
Sep 17 23:12:10.451703 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:12:10.451673    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88bdd5cd_67c8_4000_8dca_5c5875512672.slice/crio-bacfd074a3ef2a92adc95b68b37a95118f5ef80201ccf395f17123e2e0e78b31 WatchSource:0}: Error finding container bacfd074a3ef2a92adc95b68b37a95118f5ef80201ccf395f17123e2e0e78b31: Status 404 returned error can't find the container with id bacfd074a3ef2a92adc95b68b37a95118f5ef80201ccf395f17123e2e0e78b31
Sep 17 23:12:11.304824 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:11.304785    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-zd4nf" event={"ID":"88bdd5cd-67c8-4000-8dca-5c5875512672","Type":"ContainerStarted","Data":"bacfd074a3ef2a92adc95b68b37a95118f5ef80201ccf395f17123e2e0e78b31"}
Sep 17 23:12:12.314106 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:12.314068    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-zd4nf" event={"ID":"88bdd5cd-67c8-4000-8dca-5c5875512672","Type":"ContainerStarted","Data":"9fdd4eb854edc7bba861184f8dc91183e77c5939d1fc9c4505ad89320522b3ce"}
Sep 17 23:12:12.314561 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:12.314163    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-zd4nf" event={"ID":"88bdd5cd-67c8-4000-8dca-5c5875512672","Type":"ContainerStarted","Data":"d4f569d9ea9f4212d936f353586e154a0edde31d09532f4c5e55813486f1bece"}
Sep 17 23:12:12.584957 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:12.584871    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:12:12.775111 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:12.775078    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:12:15.574559 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:15.574524    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:12:16.013328 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:16.013290    2584 patch_prober.go:28] interesting pod/image-registry-75f6fdbdff-5rgvh container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 17 23:12:16.013495 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:16.013354    2584 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-75f6fdbdff-5rgvh" podUID="06535038-f186-42c3-b9d1-707e2c7ea285" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 23:12:16.017310 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:16.017276    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:12:16.077123 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:16.077072    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-zd4nf" podStartSLOduration=5.990810177 podStartE2EDuration="7.077055812s" totalImagesPullingTime="1.086245635s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:12:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:12:12.336264491 +0000 UTC m=+115.701924410" watchObservedRunningTime="2026-09-17 23:12:16.077055812 +0000 UTC m=+119.442715730"
Sep 17 23:12:16.077306 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:16.077259    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-7bd5d56d6-r6x7z"]
Sep 17 23:12:16.081628 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:16.081609    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-7bd5d56d6-r6x7z"
Sep 17 23:12:16.084210 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:16.084191    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Sep 17 23:12:16.085097 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:16.085073    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Sep 17 23:12:16.085414 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:16.085272    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 17 23:12:16.085414 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:16.085272    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-njzss\""
Sep 17 23:12:16.085414 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:16.085310    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 17 23:12:16.085414 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:16.085310    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 17 23:12:16.096740 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:16.096711    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-7bd5d56d6-r6x7z"]
Sep 17 23:12:16.104877 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:16.104846    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:12:16.137492 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:16.137465    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6885dc6c-af5e-49f8-8052-688cab953847-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-7bd5d56d6-r6x7z\" (UID: \"6885dc6c-af5e-49f8-8052-688cab953847\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-r6x7z"
Sep 17 23:12:16.137638 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:16.137513    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/6885dc6c-af5e-49f8-8052-688cab953847-prometheus-operator-tls\") pod \"prometheus-operator-7bd5d56d6-r6x7z\" (UID: \"6885dc6c-af5e-49f8-8052-688cab953847\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-r6x7z"
Sep 17 23:12:16.137638 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:16.137535    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6885dc6c-af5e-49f8-8052-688cab953847-metrics-client-ca\") pod \"prometheus-operator-7bd5d56d6-r6x7z\" (UID: \"6885dc6c-af5e-49f8-8052-688cab953847\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-r6x7z"
Sep 17 23:12:16.137638 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:16.137628    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kz5s8\" (UniqueName: \"kubernetes.io/projected/6885dc6c-af5e-49f8-8052-688cab953847-kube-api-access-kz5s8\") pod \"prometheus-operator-7bd5d56d6-r6x7z\" (UID: \"6885dc6c-af5e-49f8-8052-688cab953847\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-r6x7z"
Sep 17 23:12:16.238878 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:16.238837    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6885dc6c-af5e-49f8-8052-688cab953847-metrics-client-ca\") pod \"prometheus-operator-7bd5d56d6-r6x7z\" (UID: \"6885dc6c-af5e-49f8-8052-688cab953847\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-r6x7z"
Sep 17 23:12:16.239089 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:16.238923    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kz5s8\" (UniqueName: \"kubernetes.io/projected/6885dc6c-af5e-49f8-8052-688cab953847-kube-api-access-kz5s8\") pod \"prometheus-operator-7bd5d56d6-r6x7z\" (UID: \"6885dc6c-af5e-49f8-8052-688cab953847\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-r6x7z"
Sep 17 23:12:16.239089 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:16.238993    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6885dc6c-af5e-49f8-8052-688cab953847-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-7bd5d56d6-r6x7z\" (UID: \"6885dc6c-af5e-49f8-8052-688cab953847\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-r6x7z"
Sep 17 23:12:16.239089 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:16.239048    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/6885dc6c-af5e-49f8-8052-688cab953847-prometheus-operator-tls\") pod \"prometheus-operator-7bd5d56d6-r6x7z\" (UID: \"6885dc6c-af5e-49f8-8052-688cab953847\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-r6x7z"
Sep 17 23:12:16.239208 ip-10-0-142-4 kubenswrapper[2584]: E0917 23:12:16.239135    2584 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found
Sep 17 23:12:16.239208 ip-10-0-142-4 kubenswrapper[2584]: E0917 23:12:16.239202    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6885dc6c-af5e-49f8-8052-688cab953847-prometheus-operator-tls podName:6885dc6c-af5e-49f8-8052-688cab953847 nodeName:}" failed. No retries permitted until 2026-09-17 23:12:16.739183891 +0000 UTC m=+120.104843800 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/6885dc6c-af5e-49f8-8052-688cab953847-prometheus-operator-tls") pod "prometheus-operator-7bd5d56d6-r6x7z" (UID: "6885dc6c-af5e-49f8-8052-688cab953847") : secret "prometheus-operator-tls" not found
Sep 17 23:12:16.239572 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:16.239542    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6885dc6c-af5e-49f8-8052-688cab953847-metrics-client-ca\") pod \"prometheus-operator-7bd5d56d6-r6x7z\" (UID: \"6885dc6c-af5e-49f8-8052-688cab953847\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-r6x7z"
Sep 17 23:12:16.241814 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:16.241792    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6885dc6c-af5e-49f8-8052-688cab953847-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-7bd5d56d6-r6x7z\" (UID: \"6885dc6c-af5e-49f8-8052-688cab953847\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-r6x7z"
Sep 17 23:12:16.252581 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:16.252544    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kz5s8\" (UniqueName: \"kubernetes.io/projected/6885dc6c-af5e-49f8-8052-688cab953847-kube-api-access-kz5s8\") pod \"prometheus-operator-7bd5d56d6-r6x7z\" (UID: \"6885dc6c-af5e-49f8-8052-688cab953847\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-r6x7z"
Sep 17 23:12:16.577284 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:16.577245    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:12:16.742701 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:16.742661    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/6885dc6c-af5e-49f8-8052-688cab953847-prometheus-operator-tls\") pod \"prometheus-operator-7bd5d56d6-r6x7z\" (UID: \"6885dc6c-af5e-49f8-8052-688cab953847\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-r6x7z"
Sep 17 23:12:16.745640 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:16.745610    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/6885dc6c-af5e-49f8-8052-688cab953847-prometheus-operator-tls\") pod \"prometheus-operator-7bd5d56d6-r6x7z\" (UID: \"6885dc6c-af5e-49f8-8052-688cab953847\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-r6x7z"
Sep 17 23:12:16.992869 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:16.992792    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-7bd5d56d6-r6x7z"
Sep 17 23:12:18.019231 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:18.019184    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:12:18.119105 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:18.119066    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:12:18.234514 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:18.234483    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-75f6fdbdff-5rgvh"
Sep 17 23:12:18.503706 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:18.503673    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:12:18.576571 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:18.576536    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:12:18.940645 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:18.940565    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:12:19.019323 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:19.019290    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:12:19.302473 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:19.302437    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:12:19.404142 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:19.404102    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:12:22.350095 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:22.350063    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6594f4c75-c9vsj" event={"ID":"72436ae5-af59-40fa-8525-c27cf3209e34","Type":"ContainerStarted","Data":"35cb494fc3cdcec57857ab5d4b63a67b812fbfe612541d3099c1aa3838ef6904"}
Sep 17 23:12:22.350414 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:22.350259    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-6594f4c75-c9vsj"
Sep 17 23:12:22.351749 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:22.351728    2584 patch_prober.go:28] interesting pod/downloads-6594f4c75-c9vsj container/download-server namespace/openshift-console: Readiness probe status=failure output="Get \"http://10.132.0.12:8080/\": dial tcp 10.132.0.12:8080: connect: connection refused" start-of-body=
Sep 17 23:12:22.351790 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:22.351774    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-console/downloads-6594f4c75-c9vsj" podUID="72436ae5-af59-40fa-8525-c27cf3209e34" containerName="download-server" probeResult="failure" output="Get \"http://10.132.0.12:8080/\": dial tcp 10.132.0.12:8080: connect: connection refused"
Sep 17 23:12:22.422402 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:22.422313    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-6594f4c75-c9vsj" podStartSLOduration=1.343479261 podStartE2EDuration="17.422291462s" totalImagesPullingTime="16.078812201s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:12:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:12:22.371662207 +0000 UTC m=+125.737322124" watchObservedRunningTime="2026-09-17 23:12:22.422291462 +0000 UTC m=+125.787951381"
Sep 17 23:12:22.423497 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:22.423408    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-7bd5d56d6-r6x7z"]
Sep 17 23:12:22.425769 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:12:22.425735    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6885dc6c_af5e_49f8_8052_688cab953847.slice/crio-5ef9f36e14f91363569c0e264f74180432385a6b811667b78d3b14bbe14ed17e WatchSource:0}: Error finding container 5ef9f36e14f91363569c0e264f74180432385a6b811667b78d3b14bbe14ed17e: Status 404 returned error can't find the container with id 5ef9f36e14f91363569c0e264f74180432385a6b811667b78d3b14bbe14ed17e
Sep 17 23:12:23.338188 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:23.338142    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:12:23.356754 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:23.356672    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-7bd5d56d6-r6x7z" event={"ID":"6885dc6c-af5e-49f8-8052-688cab953847","Type":"ContainerStarted","Data":"5ef9f36e14f91363569c0e264f74180432385a6b811667b78d3b14bbe14ed17e"}
Sep 17 23:12:23.367108 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:23.367058    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-6594f4c75-c9vsj"
Sep 17 23:12:23.439438 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:23.439362    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:12:23.836966 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:23.836939    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-75f6fdbdff-5rgvh"]
Sep 17 23:12:23.992609 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:23.992529    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:12:24.094308 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:24.094271    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:12:24.361920 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:24.361872    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-7bd5d56d6-r6x7z" event={"ID":"6885dc6c-af5e-49f8-8052-688cab953847","Type":"ContainerStarted","Data":"54a6e6479bf2c87518bf329db0f3ce959a29848804d44d287b69a0c9b36915e7"}
Sep 17 23:12:24.362406 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:24.361943    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-7bd5d56d6-r6x7z" event={"ID":"6885dc6c-af5e-49f8-8052-688cab953847","Type":"ContainerStarted","Data":"957f231a4108ab4f6e1e610b087c88586d396a47927977345baf2f5faeb37170"}
Sep 17 23:12:24.386392 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:24.385060    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-7bd5d56d6-r6x7z" podStartSLOduration=7.103131919 podStartE2EDuration="8.385040123s" totalImagesPullingTime="1.281908204s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:12:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:12:24.38488001 +0000 UTC m=+127.750539928" watchObservedRunningTime="2026-09-17 23:12:24.385040123 +0000 UTC m=+127.750700041"
Sep 17 23:12:26.384231 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:26.384191    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:12:26.673968 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:26.673891    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:12:26.811277 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:26.811240    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-84c885658c-xlwlk"]
Sep 17 23:12:26.823024 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:26.822984    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:12:26.836198 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:26.836159    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-jfd6q"]
Sep 17 23:12:26.836344 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:26.836294    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-84c885658c-xlwlk"
Sep 17 23:12:26.839104 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:26.839058    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Sep 17 23:12:26.839270 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:26.839206    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Sep 17 23:12:26.840234 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:26.840209    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Sep 17 23:12:26.843217 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:26.843197    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-rf66n\""
Sep 17 23:12:26.859153 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:26.859132    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-84c885658c-xlwlk"]
Sep 17 23:12:26.859283 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:26.859267    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-jfd6q"
Sep 17 23:12:26.864249 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:26.864230    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 17 23:12:26.864545 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:26.864520    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-tdlxl\""
Sep 17 23:12:26.864758 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:26.864741    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 17 23:12:26.866572 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:26.866550    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 17 23:12:26.929028 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:26.928927    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7b67ca5d-0b03-429d-8145-6d25e2068a89-metrics-client-ca\") pod \"node-exporter-jfd6q\" (UID: \"7b67ca5d-0b03-429d-8145-6d25e2068a89\") " pod="openshift-monitoring/node-exporter-jfd6q"
Sep 17 23:12:26.929028 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:26.928984    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tdbqq\" (UniqueName: \"kubernetes.io/projected/ee9d22ab-a36f-4603-bf09-81dd9d2663f0-kube-api-access-tdbqq\") pod \"kube-state-metrics-84c885658c-xlwlk\" (UID: \"ee9d22ab-a36f-4603-bf09-81dd9d2663f0\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-xlwlk"
Sep 17 23:12:26.929259 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:26.929092    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/7b67ca5d-0b03-429d-8145-6d25e2068a89-node-exporter-tls\") pod \"node-exporter-jfd6q\" (UID: \"7b67ca5d-0b03-429d-8145-6d25e2068a89\") " pod="openshift-monitoring/node-exporter-jfd6q"
Sep 17 23:12:26.929259 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:26.929130    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/7b67ca5d-0b03-429d-8145-6d25e2068a89-node-exporter-accelerators-collector-config\") pod \"node-exporter-jfd6q\" (UID: \"7b67ca5d-0b03-429d-8145-6d25e2068a89\") " pod="openshift-monitoring/node-exporter-jfd6q"
Sep 17 23:12:26.929259 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:26.929156    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ee9d22ab-a36f-4603-bf09-81dd9d2663f0-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-84c885658c-xlwlk\" (UID: \"ee9d22ab-a36f-4603-bf09-81dd9d2663f0\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-xlwlk"
Sep 17 23:12:26.929259 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:26.929180    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/ee9d22ab-a36f-4603-bf09-81dd9d2663f0-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-84c885658c-xlwlk\" (UID: \"ee9d22ab-a36f-4603-bf09-81dd9d2663f0\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-xlwlk"
Sep 17 23:12:26.929259 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:26.929207    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/7b67ca5d-0b03-429d-8145-6d25e2068a89-node-exporter-wtmp\") pod \"node-exporter-jfd6q\" (UID: \"7b67ca5d-0b03-429d-8145-6d25e2068a89\") " pod="openshift-monitoring/node-exporter-jfd6q"
Sep 17 23:12:26.929259 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:26.929236    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7b67ca5d-0b03-429d-8145-6d25e2068a89-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-jfd6q\" (UID: \"7b67ca5d-0b03-429d-8145-6d25e2068a89\") " pod="openshift-monitoring/node-exporter-jfd6q"
Sep 17 23:12:26.929557 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:26.929264    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/ee9d22ab-a36f-4603-bf09-81dd9d2663f0-kube-state-metrics-tls\") pod \"kube-state-metrics-84c885658c-xlwlk\" (UID: \"ee9d22ab-a36f-4603-bf09-81dd9d2663f0\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-xlwlk"
Sep 17 23:12:26.929557 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:26.929287    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/ee9d22ab-a36f-4603-bf09-81dd9d2663f0-volume-directive-shadow\") pod \"kube-state-metrics-84c885658c-xlwlk\" (UID: \"ee9d22ab-a36f-4603-bf09-81dd9d2663f0\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-xlwlk"
Sep 17 23:12:26.929557 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:26.929329    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ee9d22ab-a36f-4603-bf09-81dd9d2663f0-metrics-client-ca\") pod \"kube-state-metrics-84c885658c-xlwlk\" (UID: \"ee9d22ab-a36f-4603-bf09-81dd9d2663f0\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-xlwlk"
Sep 17 23:12:26.929557 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:26.929372    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7b67ca5d-0b03-429d-8145-6d25e2068a89-sys\") pod \"node-exporter-jfd6q\" (UID: \"7b67ca5d-0b03-429d-8145-6d25e2068a89\") " pod="openshift-monitoring/node-exporter-jfd6q"
Sep 17 23:12:26.929557 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:26.929404    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/7b67ca5d-0b03-429d-8145-6d25e2068a89-root\") pod \"node-exporter-jfd6q\" (UID: \"7b67ca5d-0b03-429d-8145-6d25e2068a89\") " pod="openshift-monitoring/node-exporter-jfd6q"
Sep 17 23:12:26.929557 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:26.929424    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/7b67ca5d-0b03-429d-8145-6d25e2068a89-node-exporter-textfile\") pod \"node-exporter-jfd6q\" (UID: \"7b67ca5d-0b03-429d-8145-6d25e2068a89\") " pod="openshift-monitoring/node-exporter-jfd6q"
Sep 17 23:12:26.929557 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:26.929463    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-86fdx\" (UniqueName: \"kubernetes.io/projected/7b67ca5d-0b03-429d-8145-6d25e2068a89-kube-api-access-86fdx\") pod \"node-exporter-jfd6q\" (UID: \"7b67ca5d-0b03-429d-8145-6d25e2068a89\") " pod="openshift-monitoring/node-exporter-jfd6q"
Sep 17 23:12:26.951210 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:26.951169    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:12:27.030505 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:27.030459    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/7b67ca5d-0b03-429d-8145-6d25e2068a89-node-exporter-tls\") pod \"node-exporter-jfd6q\" (UID: \"7b67ca5d-0b03-429d-8145-6d25e2068a89\") " pod="openshift-monitoring/node-exporter-jfd6q"
Sep 17 23:12:27.030671 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:27.030514    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/7b67ca5d-0b03-429d-8145-6d25e2068a89-node-exporter-accelerators-collector-config\") pod \"node-exporter-jfd6q\" (UID: \"7b67ca5d-0b03-429d-8145-6d25e2068a89\") " pod="openshift-monitoring/node-exporter-jfd6q"
Sep 17 23:12:27.030671 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:27.030542    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ee9d22ab-a36f-4603-bf09-81dd9d2663f0-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-84c885658c-xlwlk\" (UID: \"ee9d22ab-a36f-4603-bf09-81dd9d2663f0\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-xlwlk"
Sep 17 23:12:27.030671 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:27.030566    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/ee9d22ab-a36f-4603-bf09-81dd9d2663f0-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-84c885658c-xlwlk\" (UID: \"ee9d22ab-a36f-4603-bf09-81dd9d2663f0\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-xlwlk"
Sep 17 23:12:27.030671 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:27.030594    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/7b67ca5d-0b03-429d-8145-6d25e2068a89-node-exporter-wtmp\") pod \"node-exporter-jfd6q\" (UID: \"7b67ca5d-0b03-429d-8145-6d25e2068a89\") " pod="openshift-monitoring/node-exporter-jfd6q"
Sep 17 23:12:27.030671 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:27.030618    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7b67ca5d-0b03-429d-8145-6d25e2068a89-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-jfd6q\" (UID: \"7b67ca5d-0b03-429d-8145-6d25e2068a89\") " pod="openshift-monitoring/node-exporter-jfd6q"
Sep 17 23:12:27.030671 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:27.030645    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/ee9d22ab-a36f-4603-bf09-81dd9d2663f0-kube-state-metrics-tls\") pod \"kube-state-metrics-84c885658c-xlwlk\" (UID: \"ee9d22ab-a36f-4603-bf09-81dd9d2663f0\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-xlwlk"
Sep 17 23:12:27.030671 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:27.030667    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/ee9d22ab-a36f-4603-bf09-81dd9d2663f0-volume-directive-shadow\") pod \"kube-state-metrics-84c885658c-xlwlk\" (UID: \"ee9d22ab-a36f-4603-bf09-81dd9d2663f0\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-xlwlk"
Sep 17 23:12:27.031072 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:27.030689    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ee9d22ab-a36f-4603-bf09-81dd9d2663f0-metrics-client-ca\") pod \"kube-state-metrics-84c885658c-xlwlk\" (UID: \"ee9d22ab-a36f-4603-bf09-81dd9d2663f0\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-xlwlk"
Sep 17 23:12:27.031072 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:27.030715    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7b67ca5d-0b03-429d-8145-6d25e2068a89-sys\") pod \"node-exporter-jfd6q\" (UID: \"7b67ca5d-0b03-429d-8145-6d25e2068a89\") " pod="openshift-monitoring/node-exporter-jfd6q"
Sep 17 23:12:27.031072 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:27.030741    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/7b67ca5d-0b03-429d-8145-6d25e2068a89-root\") pod \"node-exporter-jfd6q\" (UID: \"7b67ca5d-0b03-429d-8145-6d25e2068a89\") " pod="openshift-monitoring/node-exporter-jfd6q"
Sep 17 23:12:27.031072 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:27.030767    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/7b67ca5d-0b03-429d-8145-6d25e2068a89-node-exporter-textfile\") pod \"node-exporter-jfd6q\" (UID: \"7b67ca5d-0b03-429d-8145-6d25e2068a89\") " pod="openshift-monitoring/node-exporter-jfd6q"
Sep 17 23:12:27.031072 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:27.030775    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/7b67ca5d-0b03-429d-8145-6d25e2068a89-node-exporter-wtmp\") pod \"node-exporter-jfd6q\" (UID: \"7b67ca5d-0b03-429d-8145-6d25e2068a89\") " pod="openshift-monitoring/node-exporter-jfd6q"
Sep 17 23:12:27.031072 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:27.030792    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86fdx\" (UniqueName: \"kubernetes.io/projected/7b67ca5d-0b03-429d-8145-6d25e2068a89-kube-api-access-86fdx\") pod \"node-exporter-jfd6q\" (UID: \"7b67ca5d-0b03-429d-8145-6d25e2068a89\") " pod="openshift-monitoring/node-exporter-jfd6q"
Sep 17 23:12:27.031072 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:27.030873    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7b67ca5d-0b03-429d-8145-6d25e2068a89-metrics-client-ca\") pod \"node-exporter-jfd6q\" (UID: \"7b67ca5d-0b03-429d-8145-6d25e2068a89\") " pod="openshift-monitoring/node-exporter-jfd6q"
Sep 17 23:12:27.031072 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:27.030922    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tdbqq\" (UniqueName: \"kubernetes.io/projected/ee9d22ab-a36f-4603-bf09-81dd9d2663f0-kube-api-access-tdbqq\") pod \"kube-state-metrics-84c885658c-xlwlk\" (UID: \"ee9d22ab-a36f-4603-bf09-81dd9d2663f0\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-xlwlk"
Sep 17 23:12:27.031467 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:27.031211    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/7b67ca5d-0b03-429d-8145-6d25e2068a89-node-exporter-accelerators-collector-config\") pod \"node-exporter-jfd6q\" (UID: \"7b67ca5d-0b03-429d-8145-6d25e2068a89\") " pod="openshift-monitoring/node-exporter-jfd6q"
Sep 17 23:12:27.031467 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:27.031281    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/7b67ca5d-0b03-429d-8145-6d25e2068a89-root\") pod \"node-exporter-jfd6q\" (UID: \"7b67ca5d-0b03-429d-8145-6d25e2068a89\") " pod="openshift-monitoring/node-exporter-jfd6q"
Sep 17 23:12:27.031467 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:27.031328    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7b67ca5d-0b03-429d-8145-6d25e2068a89-sys\") pod \"node-exporter-jfd6q\" (UID: \"7b67ca5d-0b03-429d-8145-6d25e2068a89\") " pod="openshift-monitoring/node-exporter-jfd6q"
Sep 17 23:12:27.031705 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:27.031679    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/ee9d22ab-a36f-4603-bf09-81dd9d2663f0-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-84c885658c-xlwlk\" (UID: \"ee9d22ab-a36f-4603-bf09-81dd9d2663f0\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-xlwlk"
Sep 17 23:12:27.031803 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:27.031782    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7b67ca5d-0b03-429d-8145-6d25e2068a89-metrics-client-ca\") pod \"node-exporter-jfd6q\" (UID: \"7b67ca5d-0b03-429d-8145-6d25e2068a89\") " pod="openshift-monitoring/node-exporter-jfd6q"
Sep 17 23:12:27.032080 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:27.031901    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/ee9d22ab-a36f-4603-bf09-81dd9d2663f0-volume-directive-shadow\") pod \"kube-state-metrics-84c885658c-xlwlk\" (UID: \"ee9d22ab-a36f-4603-bf09-81dd9d2663f0\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-xlwlk"
Sep 17 23:12:27.032080 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:27.031968    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/7b67ca5d-0b03-429d-8145-6d25e2068a89-node-exporter-textfile\") pod \"node-exporter-jfd6q\" (UID: \"7b67ca5d-0b03-429d-8145-6d25e2068a89\") " pod="openshift-monitoring/node-exporter-jfd6q"
Sep 17 23:12:27.032080 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:27.031991    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ee9d22ab-a36f-4603-bf09-81dd9d2663f0-metrics-client-ca\") pod \"kube-state-metrics-84c885658c-xlwlk\" (UID: \"ee9d22ab-a36f-4603-bf09-81dd9d2663f0\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-xlwlk"
Sep 17 23:12:27.033856 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:27.033828    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ee9d22ab-a36f-4603-bf09-81dd9d2663f0-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-84c885658c-xlwlk\" (UID: \"ee9d22ab-a36f-4603-bf09-81dd9d2663f0\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-xlwlk"
Sep 17 23:12:27.033997 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:27.033975    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7b67ca5d-0b03-429d-8145-6d25e2068a89-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-jfd6q\" (UID: \"7b67ca5d-0b03-429d-8145-6d25e2068a89\") " pod="openshift-monitoring/node-exporter-jfd6q"
Sep 17 23:12:27.034466 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:27.034438    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/7b67ca5d-0b03-429d-8145-6d25e2068a89-node-exporter-tls\") pod \"node-exporter-jfd6q\" (UID: \"7b67ca5d-0b03-429d-8145-6d25e2068a89\") " pod="openshift-monitoring/node-exporter-jfd6q"
Sep 17 23:12:27.034627 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:27.034599    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/ee9d22ab-a36f-4603-bf09-81dd9d2663f0-kube-state-metrics-tls\") pod \"kube-state-metrics-84c885658c-xlwlk\" (UID: \"ee9d22ab-a36f-4603-bf09-81dd9d2663f0\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-xlwlk"
Sep 17 23:12:27.041584 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:27.041557    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tdbqq\" (UniqueName: \"kubernetes.io/projected/ee9d22ab-a36f-4603-bf09-81dd9d2663f0-kube-api-access-tdbqq\") pod \"kube-state-metrics-84c885658c-xlwlk\" (UID: \"ee9d22ab-a36f-4603-bf09-81dd9d2663f0\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-xlwlk"
Sep 17 23:12:27.041966 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:27.041950    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-86fdx\" (UniqueName: \"kubernetes.io/projected/7b67ca5d-0b03-429d-8145-6d25e2068a89-kube-api-access-86fdx\") pod \"node-exporter-jfd6q\" (UID: \"7b67ca5d-0b03-429d-8145-6d25e2068a89\") " pod="openshift-monitoring/node-exporter-jfd6q"
Sep 17 23:12:27.116111 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:27.116064    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:12:27.149098 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:27.149068    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-84c885658c-xlwlk"
Sep 17 23:12:27.169285 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:27.169251    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-jfd6q"
Sep 17 23:12:27.222347 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:12:27.222298    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b67ca5d_0b03_429d_8145_6d25e2068a89.slice/crio-66a6af452f985d81bd833ac0c1749a19fc30927524a1bc823de855d206b0bf5a WatchSource:0}: Error finding container 66a6af452f985d81bd833ac0c1749a19fc30927524a1bc823de855d206b0bf5a: Status 404 returned error can't find the container with id 66a6af452f985d81bd833ac0c1749a19fc30927524a1bc823de855d206b0bf5a
Sep 17 23:12:27.228402 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:27.228359    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:12:27.334600 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:27.334561    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:12:27.374264 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:27.374228    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-jfd6q" event={"ID":"7b67ca5d-0b03-429d-8145-6d25e2068a89","Type":"ContainerStarted","Data":"66a6af452f985d81bd833ac0c1749a19fc30927524a1bc823de855d206b0bf5a"}
Sep 17 23:12:27.473203 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:27.473128    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-84c885658c-xlwlk"]
Sep 17 23:12:27.476416 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:12:27.476386    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee9d22ab_a36f_4603_bf09_81dd9d2663f0.slice/crio-34792f5c81dde2501c1b4959f6ec75f2984ea8439e87a5e13cd2a4ab96f1db11 WatchSource:0}: Error finding container 34792f5c81dde2501c1b4959f6ec75f2984ea8439e87a5e13cd2a4ab96f1db11: Status 404 returned error can't find the container with id 34792f5c81dde2501c1b4959f6ec75f2984ea8439e87a5e13cd2a4ab96f1db11
Sep 17 23:12:27.645808 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:27.645782    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:12:27.745808 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:27.745685    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:12:27.779212 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:27.779179    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 17 23:12:27.795239 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:27.795204    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 17 23:12:27.798852 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:27.798740    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Sep 17 23:12:27.799282 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:27.799262    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Sep 17 23:12:27.799476 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:27.799448    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Sep 17 23:12:27.799614 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:27.799596    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Sep 17 23:12:27.799781 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:27.799764    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Sep 17 23:12:27.800180 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:27.800159    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Sep 17 23:12:27.800273 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:27.800236    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Sep 17 23:12:27.800548 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:27.800395    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Sep 17 23:12:27.801553 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:27.801531    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Sep 17 23:12:27.803385 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:27.803309    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-w8b6x\""
Sep 17 23:12:27.810736 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:27.810650    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 17 23:12:27.838217 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:27.837587    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/9d77d57a-58f9-4d62-baef-c4cb8b62816f-config-volume\") pod \"alertmanager-main-0\" (UID: \"9d77d57a-58f9-4d62-baef-c4cb8b62816f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 23:12:27.838217 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:27.837643    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/9d77d57a-58f9-4d62-baef-c4cb8b62816f-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"9d77d57a-58f9-4d62-baef-c4cb8b62816f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 23:12:27.838217 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:27.837675    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9d77d57a-58f9-4d62-baef-c4cb8b62816f-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"9d77d57a-58f9-4d62-baef-c4cb8b62816f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 23:12:27.838217 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:27.837698    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-87trb\" (UniqueName: \"kubernetes.io/projected/9d77d57a-58f9-4d62-baef-c4cb8b62816f-kube-api-access-87trb\") pod \"alertmanager-main-0\" (UID: \"9d77d57a-58f9-4d62-baef-c4cb8b62816f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 23:12:27.838217 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:27.837739    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9d77d57a-58f9-4d62-baef-c4cb8b62816f-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"9d77d57a-58f9-4d62-baef-c4cb8b62816f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 23:12:27.838217 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:27.837774    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9d77d57a-58f9-4d62-baef-c4cb8b62816f-tls-assets\") pod \"alertmanager-main-0\" (UID: \"9d77d57a-58f9-4d62-baef-c4cb8b62816f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 23:12:27.838217 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:27.837799    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9d77d57a-58f9-4d62-baef-c4cb8b62816f-web-config\") pod \"alertmanager-main-0\" (UID: \"9d77d57a-58f9-4d62-baef-c4cb8b62816f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 23:12:27.838217 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:27.837828    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/9d77d57a-58f9-4d62-baef-c4cb8b62816f-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"9d77d57a-58f9-4d62-baef-c4cb8b62816f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 23:12:27.838217 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:27.837878    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9d77d57a-58f9-4d62-baef-c4cb8b62816f-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"9d77d57a-58f9-4d62-baef-c4cb8b62816f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 23:12:27.838217 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:27.837905    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/9d77d57a-58f9-4d62-baef-c4cb8b62816f-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"9d77d57a-58f9-4d62-baef-c4cb8b62816f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 23:12:27.838217 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:27.837933    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/9d77d57a-58f9-4d62-baef-c4cb8b62816f-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"9d77d57a-58f9-4d62-baef-c4cb8b62816f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 23:12:27.838217 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:27.837985    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9d77d57a-58f9-4d62-baef-c4cb8b62816f-config-out\") pod \"alertmanager-main-0\" (UID: \"9d77d57a-58f9-4d62-baef-c4cb8b62816f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 23:12:27.838217 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:27.838027    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9d77d57a-58f9-4d62-baef-c4cb8b62816f-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"9d77d57a-58f9-4d62-baef-c4cb8b62816f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 23:12:27.888538 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:27.888503    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:12:27.939234 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:27.939195    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/9d77d57a-58f9-4d62-baef-c4cb8b62816f-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"9d77d57a-58f9-4d62-baef-c4cb8b62816f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 23:12:27.939442 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:27.939242    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9d77d57a-58f9-4d62-baef-c4cb8b62816f-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"9d77d57a-58f9-4d62-baef-c4cb8b62816f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 23:12:27.939442 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:27.939277    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87trb\" (UniqueName: \"kubernetes.io/projected/9d77d57a-58f9-4d62-baef-c4cb8b62816f-kube-api-access-87trb\") pod \"alertmanager-main-0\" (UID: \"9d77d57a-58f9-4d62-baef-c4cb8b62816f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 23:12:27.939442 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:27.939312    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9d77d57a-58f9-4d62-baef-c4cb8b62816f-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"9d77d57a-58f9-4d62-baef-c4cb8b62816f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 23:12:27.939442 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:27.939339    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9d77d57a-58f9-4d62-baef-c4cb8b62816f-tls-assets\") pod \"alertmanager-main-0\" (UID: \"9d77d57a-58f9-4d62-baef-c4cb8b62816f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 23:12:27.939442 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:27.939364    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9d77d57a-58f9-4d62-baef-c4cb8b62816f-web-config\") pod \"alertmanager-main-0\" (UID: \"9d77d57a-58f9-4d62-baef-c4cb8b62816f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 23:12:27.939442 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:27.939389    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/9d77d57a-58f9-4d62-baef-c4cb8b62816f-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"9d77d57a-58f9-4d62-baef-c4cb8b62816f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 23:12:27.939442 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:27.939421    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9d77d57a-58f9-4d62-baef-c4cb8b62816f-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"9d77d57a-58f9-4d62-baef-c4cb8b62816f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 23:12:27.939751 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:27.939461    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/9d77d57a-58f9-4d62-baef-c4cb8b62816f-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"9d77d57a-58f9-4d62-baef-c4cb8b62816f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 23:12:27.939751 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:27.939482    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/9d77d57a-58f9-4d62-baef-c4cb8b62816f-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"9d77d57a-58f9-4d62-baef-c4cb8b62816f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 23:12:27.939751 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:27.939526    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9d77d57a-58f9-4d62-baef-c4cb8b62816f-config-out\") pod \"alertmanager-main-0\" (UID: \"9d77d57a-58f9-4d62-baef-c4cb8b62816f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 23:12:27.939751 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:27.939545    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9d77d57a-58f9-4d62-baef-c4cb8b62816f-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"9d77d57a-58f9-4d62-baef-c4cb8b62816f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 23:12:27.939751 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:27.939606    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/9d77d57a-58f9-4d62-baef-c4cb8b62816f-config-volume\") pod \"alertmanager-main-0\" (UID: \"9d77d57a-58f9-4d62-baef-c4cb8b62816f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 23:12:27.940956 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:27.940930    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9d77d57a-58f9-4d62-baef-c4cb8b62816f-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"9d77d57a-58f9-4d62-baef-c4cb8b62816f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 23:12:27.941775 ip-10-0-142-4 kubenswrapper[2584]: E0917 23:12:27.941752    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9d77d57a-58f9-4d62-baef-c4cb8b62816f-alertmanager-trusted-ca-bundle podName:9d77d57a-58f9-4d62-baef-c4cb8b62816f nodeName:}" failed. No retries permitted until 2026-09-17 23:12:28.441732499 +0000 UTC m=+131.807392394 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "alertmanager-trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/9d77d57a-58f9-4d62-baef-c4cb8b62816f-alertmanager-trusted-ca-bundle") pod "alertmanager-main-0" (UID: "9d77d57a-58f9-4d62-baef-c4cb8b62816f") : configmap references non-existent config key: ca-bundle.crt
Sep 17 23:12:27.943728 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:27.943671    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/9d77d57a-58f9-4d62-baef-c4cb8b62816f-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"9d77d57a-58f9-4d62-baef-c4cb8b62816f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 23:12:27.944848 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:27.944819    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9d77d57a-58f9-4d62-baef-c4cb8b62816f-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"9d77d57a-58f9-4d62-baef-c4cb8b62816f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 23:12:27.945963 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:27.945937    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9d77d57a-58f9-4d62-baef-c4cb8b62816f-tls-assets\") pod \"alertmanager-main-0\" (UID: \"9d77d57a-58f9-4d62-baef-c4cb8b62816f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 23:12:27.945963 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:27.945958    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9d77d57a-58f9-4d62-baef-c4cb8b62816f-web-config\") pod \"alertmanager-main-0\" (UID: \"9d77d57a-58f9-4d62-baef-c4cb8b62816f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 23:12:27.946156 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:27.945940    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/9d77d57a-58f9-4d62-baef-c4cb8b62816f-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"9d77d57a-58f9-4d62-baef-c4cb8b62816f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 23:12:27.946403 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:27.946381    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9d77d57a-58f9-4d62-baef-c4cb8b62816f-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"9d77d57a-58f9-4d62-baef-c4cb8b62816f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 23:12:27.947637 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:27.947587    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/9d77d57a-58f9-4d62-baef-c4cb8b62816f-config-volume\") pod \"alertmanager-main-0\" (UID: \"9d77d57a-58f9-4d62-baef-c4cb8b62816f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 23:12:27.947957 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:27.947911    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9d77d57a-58f9-4d62-baef-c4cb8b62816f-config-out\") pod \"alertmanager-main-0\" (UID: \"9d77d57a-58f9-4d62-baef-c4cb8b62816f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 23:12:27.948375 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:27.948349    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/9d77d57a-58f9-4d62-baef-c4cb8b62816f-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"9d77d57a-58f9-4d62-baef-c4cb8b62816f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 23:12:27.948486 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:27.948390    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/9d77d57a-58f9-4d62-baef-c4cb8b62816f-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"9d77d57a-58f9-4d62-baef-c4cb8b62816f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 23:12:27.958422 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:27.958396    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-87trb\" (UniqueName: \"kubernetes.io/projected/9d77d57a-58f9-4d62-baef-c4cb8b62816f-kube-api-access-87trb\") pod \"alertmanager-main-0\" (UID: \"9d77d57a-58f9-4d62-baef-c4cb8b62816f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 23:12:28.082082 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:28.082029    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:12:28.184173 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:28.184140    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:12:28.381846 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:28.381703    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-84c885658c-xlwlk" event={"ID":"ee9d22ab-a36f-4603-bf09-81dd9d2663f0","Type":"ContainerStarted","Data":"34792f5c81dde2501c1b4959f6ec75f2984ea8439e87a5e13cd2a4ab96f1db11"}
Sep 17 23:12:28.444671 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:28.444631    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9d77d57a-58f9-4d62-baef-c4cb8b62816f-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"9d77d57a-58f9-4d62-baef-c4cb8b62816f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 23:12:28.445798 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:28.445772    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9d77d57a-58f9-4d62-baef-c4cb8b62816f-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"9d77d57a-58f9-4d62-baef-c4cb8b62816f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 23:12:28.546117 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:28.546086    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d672140f-ad59-4a68-9dde-80b62492af7f-metrics-certs\") pod \"network-metrics-daemon-4bbdp\" (UID: \"d672140f-ad59-4a68-9dde-80b62492af7f\") " pod="openshift-multus/network-metrics-daemon-4bbdp"
Sep 17 23:12:28.548900 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:28.548877    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d672140f-ad59-4a68-9dde-80b62492af7f-metrics-certs\") pod \"network-metrics-daemon-4bbdp\" (UID: \"d672140f-ad59-4a68-9dde-80b62492af7f\") " pod="openshift-multus/network-metrics-daemon-4bbdp"
Sep 17 23:12:28.728723 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:28.728619    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 17 23:12:28.776356 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:28.776331    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-k7845\""
Sep 17 23:12:28.784076 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:28.784049    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4bbdp"
Sep 17 23:12:28.790331 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:28.789643    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-6c99487fbd-897xp"]
Sep 17 23:12:28.807127 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:28.807103    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-6c99487fbd-897xp"
Sep 17 23:12:28.810538 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:28.810311    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Sep 17 23:12:28.810538 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:28.810332    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Sep 17 23:12:28.810538 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:28.810513    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Sep 17 23:12:28.810746 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:28.810684    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Sep 17 23:12:28.811297 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:28.811275    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Sep 17 23:12:28.811483 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:28.811343    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-8hf5gg3dqikht\""
Sep 17 23:12:28.811610 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:28.811509    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-rjmcc\""
Sep 17 23:12:28.820709 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:28.820681    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-6c99487fbd-897xp"]
Sep 17 23:12:28.848019 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:28.847970    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e2b185f4-e371-4556-a3ae-3eda3d226876-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6c99487fbd-897xp\" (UID: \"e2b185f4-e371-4556-a3ae-3eda3d226876\") " pod="openshift-monitoring/thanos-querier-6c99487fbd-897xp"
Sep 17 23:12:28.848212 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:28.848087    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e2b185f4-e371-4556-a3ae-3eda3d226876-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6c99487fbd-897xp\" (UID: \"e2b185f4-e371-4556-a3ae-3eda3d226876\") " pod="openshift-monitoring/thanos-querier-6c99487fbd-897xp"
Sep 17 23:12:28.848212 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:28.848162    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/e2b185f4-e371-4556-a3ae-3eda3d226876-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6c99487fbd-897xp\" (UID: \"e2b185f4-e371-4556-a3ae-3eda3d226876\") " pod="openshift-monitoring/thanos-querier-6c99487fbd-897xp"
Sep 17 23:12:28.848212 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:28.848191    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/e2b185f4-e371-4556-a3ae-3eda3d226876-secret-grpc-tls\") pod \"thanos-querier-6c99487fbd-897xp\" (UID: \"e2b185f4-e371-4556-a3ae-3eda3d226876\") " pod="openshift-monitoring/thanos-querier-6c99487fbd-897xp"
Sep 17 23:12:28.848373 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:28.848242    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/e2b185f4-e371-4556-a3ae-3eda3d226876-secret-thanos-querier-tls\") pod \"thanos-querier-6c99487fbd-897xp\" (UID: \"e2b185f4-e371-4556-a3ae-3eda3d226876\") " pod="openshift-monitoring/thanos-querier-6c99487fbd-897xp"
Sep 17 23:12:28.848489 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:28.848415    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/e2b185f4-e371-4556-a3ae-3eda3d226876-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6c99487fbd-897xp\" (UID: \"e2b185f4-e371-4556-a3ae-3eda3d226876\") " pod="openshift-monitoring/thanos-querier-6c99487fbd-897xp"
Sep 17 23:12:28.848597 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:28.848572    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e2b185f4-e371-4556-a3ae-3eda3d226876-metrics-client-ca\") pod \"thanos-querier-6c99487fbd-897xp\" (UID: \"e2b185f4-e371-4556-a3ae-3eda3d226876\") " pod="openshift-monitoring/thanos-querier-6c99487fbd-897xp"
Sep 17 23:12:28.848757 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:28.848723    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h6qhm\" (UniqueName: \"kubernetes.io/projected/e2b185f4-e371-4556-a3ae-3eda3d226876-kube-api-access-h6qhm\") pod \"thanos-querier-6c99487fbd-897xp\" (UID: \"e2b185f4-e371-4556-a3ae-3eda3d226876\") " pod="openshift-monitoring/thanos-querier-6c99487fbd-897xp"
Sep 17 23:12:28.949370 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:28.949328    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/e2b185f4-e371-4556-a3ae-3eda3d226876-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6c99487fbd-897xp\" (UID: \"e2b185f4-e371-4556-a3ae-3eda3d226876\") " pod="openshift-monitoring/thanos-querier-6c99487fbd-897xp"
Sep 17 23:12:28.949370 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:28.949373    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/e2b185f4-e371-4556-a3ae-3eda3d226876-secret-grpc-tls\") pod \"thanos-querier-6c99487fbd-897xp\" (UID: \"e2b185f4-e371-4556-a3ae-3eda3d226876\") " pod="openshift-monitoring/thanos-querier-6c99487fbd-897xp"
Sep 17 23:12:28.949942 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:28.949402    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/e2b185f4-e371-4556-a3ae-3eda3d226876-secret-thanos-querier-tls\") pod \"thanos-querier-6c99487fbd-897xp\" (UID: \"e2b185f4-e371-4556-a3ae-3eda3d226876\") " pod="openshift-monitoring/thanos-querier-6c99487fbd-897xp"
Sep 17 23:12:28.949942 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:28.949435    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/e2b185f4-e371-4556-a3ae-3eda3d226876-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6c99487fbd-897xp\" (UID: \"e2b185f4-e371-4556-a3ae-3eda3d226876\") " pod="openshift-monitoring/thanos-querier-6c99487fbd-897xp"
Sep 17 23:12:28.949942 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:28.949474    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e2b185f4-e371-4556-a3ae-3eda3d226876-metrics-client-ca\") pod \"thanos-querier-6c99487fbd-897xp\" (UID: \"e2b185f4-e371-4556-a3ae-3eda3d226876\") " pod="openshift-monitoring/thanos-querier-6c99487fbd-897xp"
Sep 17 23:12:28.949942 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:28.949503    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h6qhm\" (UniqueName: \"kubernetes.io/projected/e2b185f4-e371-4556-a3ae-3eda3d226876-kube-api-access-h6qhm\") pod \"thanos-querier-6c99487fbd-897xp\" (UID: \"e2b185f4-e371-4556-a3ae-3eda3d226876\") " pod="openshift-monitoring/thanos-querier-6c99487fbd-897xp"
Sep 17 23:12:28.949942 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:28.949654    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e2b185f4-e371-4556-a3ae-3eda3d226876-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6c99487fbd-897xp\" (UID: \"e2b185f4-e371-4556-a3ae-3eda3d226876\") " pod="openshift-monitoring/thanos-querier-6c99487fbd-897xp"
Sep 17 23:12:28.949942 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:28.949722    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e2b185f4-e371-4556-a3ae-3eda3d226876-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6c99487fbd-897xp\" (UID: \"e2b185f4-e371-4556-a3ae-3eda3d226876\") " pod="openshift-monitoring/thanos-querier-6c99487fbd-897xp"
Sep 17 23:12:28.950374 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:28.950259    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e2b185f4-e371-4556-a3ae-3eda3d226876-metrics-client-ca\") pod \"thanos-querier-6c99487fbd-897xp\" (UID: \"e2b185f4-e371-4556-a3ae-3eda3d226876\") " pod="openshift-monitoring/thanos-querier-6c99487fbd-897xp"
Sep 17 23:12:28.953074 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:28.953050    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/e2b185f4-e371-4556-a3ae-3eda3d226876-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6c99487fbd-897xp\" (UID: \"e2b185f4-e371-4556-a3ae-3eda3d226876\") " pod="openshift-monitoring/thanos-querier-6c99487fbd-897xp"
Sep 17 23:12:28.953755 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:28.953713    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e2b185f4-e371-4556-a3ae-3eda3d226876-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6c99487fbd-897xp\" (UID: \"e2b185f4-e371-4556-a3ae-3eda3d226876\") " pod="openshift-monitoring/thanos-querier-6c99487fbd-897xp"
Sep 17 23:12:28.953755 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:28.953745    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e2b185f4-e371-4556-a3ae-3eda3d226876-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6c99487fbd-897xp\" (UID: \"e2b185f4-e371-4556-a3ae-3eda3d226876\") " pod="openshift-monitoring/thanos-querier-6c99487fbd-897xp"
Sep 17 23:12:28.953899 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:28.953760    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/e2b185f4-e371-4556-a3ae-3eda3d226876-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6c99487fbd-897xp\" (UID: \"e2b185f4-e371-4556-a3ae-3eda3d226876\") " pod="openshift-monitoring/thanos-querier-6c99487fbd-897xp"
Sep 17 23:12:28.954423 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:28.954405    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/e2b185f4-e371-4556-a3ae-3eda3d226876-secret-thanos-querier-tls\") pod \"thanos-querier-6c99487fbd-897xp\" (UID: \"e2b185f4-e371-4556-a3ae-3eda3d226876\") " pod="openshift-monitoring/thanos-querier-6c99487fbd-897xp"
Sep 17 23:12:28.954975 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:28.954956    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/e2b185f4-e371-4556-a3ae-3eda3d226876-secret-grpc-tls\") pod \"thanos-querier-6c99487fbd-897xp\" (UID: \"e2b185f4-e371-4556-a3ae-3eda3d226876\") " pod="openshift-monitoring/thanos-querier-6c99487fbd-897xp"
Sep 17 23:12:28.959701 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:28.959680    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h6qhm\" (UniqueName: \"kubernetes.io/projected/e2b185f4-e371-4556-a3ae-3eda3d226876-kube-api-access-h6qhm\") pod \"thanos-querier-6c99487fbd-897xp\" (UID: \"e2b185f4-e371-4556-a3ae-3eda3d226876\") " pod="openshift-monitoring/thanos-querier-6c99487fbd-897xp"
Sep 17 23:12:29.121747 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:29.121705    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-6c99487fbd-897xp"
Sep 17 23:12:29.759106 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:29.759074    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-6c99487fbd-897xp"]
Sep 17 23:12:29.760994 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:12:29.760963    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2b185f4_e371_4556_a3ae_3eda3d226876.slice/crio-eda8b3ba8c5bd4eba91adddead395e4bd771ada6aebd91e17469ea32a11b61fb WatchSource:0}: Error finding container eda8b3ba8c5bd4eba91adddead395e4bd771ada6aebd91e17469ea32a11b61fb: Status 404 returned error can't find the container with id eda8b3ba8c5bd4eba91adddead395e4bd771ada6aebd91e17469ea32a11b61fb
Sep 17 23:12:29.776860 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:29.776706    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-4bbdp"]
Sep 17 23:12:29.780784 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:12:29.780743    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd672140f_ad59_4a68_9dde_80b62492af7f.slice/crio-d99ef3afd5173b0336f2cf8f97c9d17e540577042c3499eb90679d50b94388bc WatchSource:0}: Error finding container d99ef3afd5173b0336f2cf8f97c9d17e540577042c3499eb90679d50b94388bc: Status 404 returned error can't find the container with id d99ef3afd5173b0336f2cf8f97c9d17e540577042c3499eb90679d50b94388bc
Sep 17 23:12:29.811051 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:29.810949    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 17 23:12:29.811339 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:12:29.811311    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d77d57a_58f9_4d62_baef_c4cb8b62816f.slice/crio-e43f3c228a3f4be9c49c399b086234b30bf9faf8ba424a4a95652d2329e4d416 WatchSource:0}: Error finding container e43f3c228a3f4be9c49c399b086234b30bf9faf8ba424a4a95652d2329e4d416: Status 404 returned error can't find the container with id e43f3c228a3f4be9c49c399b086234b30bf9faf8ba424a4a95652d2329e4d416
Sep 17 23:12:30.158165 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:30.158128    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:12:30.289855 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:30.289790    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:12:30.382030 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:30.380301    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:12:30.393859 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:30.393820    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-4bbdp" event={"ID":"d672140f-ad59-4a68-9dde-80b62492af7f","Type":"ContainerStarted","Data":"d99ef3afd5173b0336f2cf8f97c9d17e540577042c3499eb90679d50b94388bc"}
Sep 17 23:12:30.397431 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:30.397394    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9d77d57a-58f9-4d62-baef-c4cb8b62816f","Type":"ContainerStarted","Data":"e43f3c228a3f4be9c49c399b086234b30bf9faf8ba424a4a95652d2329e4d416"}
Sep 17 23:12:30.402258 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:30.402218    2584 generic.go:356] "Generic (PLEG): container finished" podID="7b67ca5d-0b03-429d-8145-6d25e2068a89" containerID="f3fcea6b764d53b1b3eeef0ba94cf8db4f3a795be9d9ccb546815ef6c8e64e35" exitCode=0
Sep 17 23:12:30.402501 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:30.402335    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-jfd6q" event={"ID":"7b67ca5d-0b03-429d-8145-6d25e2068a89","Type":"ContainerDied","Data":"f3fcea6b764d53b1b3eeef0ba94cf8db4f3a795be9d9ccb546815ef6c8e64e35"}
Sep 17 23:12:30.415289 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:30.415076    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-84c885658c-xlwlk" event={"ID":"ee9d22ab-a36f-4603-bf09-81dd9d2663f0","Type":"ContainerStarted","Data":"b0736ea360e6bd67613caec1d78443bffa7c29de5dc53d4ff31482b66b946905"}
Sep 17 23:12:30.415289 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:30.415144    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-84c885658c-xlwlk" event={"ID":"ee9d22ab-a36f-4603-bf09-81dd9d2663f0","Type":"ContainerStarted","Data":"e52e2bd491661c8b478d82656b51663d3b41d7fd1b3f37ab2e9187c9a7c4f6f7"}
Sep 17 23:12:30.415289 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:30.415181    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-84c885658c-xlwlk" event={"ID":"ee9d22ab-a36f-4603-bf09-81dd9d2663f0","Type":"ContainerStarted","Data":"891925db95acefe93e6f6826cb048142590308ef0e8ecd5077f2b20a301086b4"}
Sep 17 23:12:30.421037 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:30.420866    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6c99487fbd-897xp" event={"ID":"e2b185f4-e371-4556-a3ae-3eda3d226876","Type":"ContainerStarted","Data":"eda8b3ba8c5bd4eba91adddead395e4bd771ada6aebd91e17469ea32a11b61fb"}
Sep 17 23:12:31.333941 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:31.333874    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:12:31.417511 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:31.417472    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:12:31.431411 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:31.431301    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-jfd6q" event={"ID":"7b67ca5d-0b03-429d-8145-6d25e2068a89","Type":"ContainerStarted","Data":"8742d4e153008ac84ac1513f88613e9f406cf6322cc3ecfd708df4949028e481"}
Sep 17 23:12:31.431411 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:31.431361    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-jfd6q" event={"ID":"7b67ca5d-0b03-429d-8145-6d25e2068a89","Type":"ContainerStarted","Data":"42e51d37626cef2bafff778b71053205befa4710ef5f9b596a106fa9abebaa6a"}
Sep 17 23:12:31.457319 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:31.457261    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-84c885658c-xlwlk" podStartSLOduration=3.540866668 podStartE2EDuration="5.45723893s" totalImagesPullingTime="1.916372262s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:12:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:12:30.453204743 +0000 UTC m=+133.818864660" watchObservedRunningTime="2026-09-17 23:12:31.45723893 +0000 UTC m=+134.822898848"
Sep 17 23:12:31.457504 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:31.457396    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-jfd6q" podStartSLOduration=3.176652086 podStartE2EDuration="5.457385471s" totalImagesPullingTime="2.169564896s" totalInitContainerRuntime="111.168489ms" isStatefulPod=true podCreationTimestamp="2026-09-17 23:12:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:12:31.456181574 +0000 UTC m=+134.821841507" watchObservedRunningTime="2026-09-17 23:12:31.457385471 +0000 UTC m=+134.823045388"
Sep 17 23:12:31.757410 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:31.757378    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:12:31.867705 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:31.867672    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:12:31.963772 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:31.963735    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:12:33.440096 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:33.439985    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6c99487fbd-897xp" event={"ID":"e2b185f4-e371-4556-a3ae-3eda3d226876","Type":"ContainerStarted","Data":"e0c9fd953eb6cb8b19977faa7e4652ab54bc117639300aba0e2698de0aef718c"}
Sep 17 23:12:33.440585 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:33.440098    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6c99487fbd-897xp" event={"ID":"e2b185f4-e371-4556-a3ae-3eda3d226876","Type":"ContainerStarted","Data":"5c63ef0bab1dfb3b07fd53b71f1f801d40773f0bc0e3bbb4734794960e3b5383"}
Sep 17 23:12:33.440585 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:33.440137    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6c99487fbd-897xp" event={"ID":"e2b185f4-e371-4556-a3ae-3eda3d226876","Type":"ContainerStarted","Data":"31425f0ef8f3466f7d7a19d911cc306fb8d29dc19ebf4e7b15b9d87546d21578"}
Sep 17 23:12:33.442279 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:33.442252    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-4bbdp" event={"ID":"d672140f-ad59-4a68-9dde-80b62492af7f","Type":"ContainerStarted","Data":"8818d0e508d97dd185052f109599fb2a5e6d4f76ba22741442882b933faf9cd1"}
Sep 17 23:12:33.442404 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:33.442306    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-4bbdp" event={"ID":"d672140f-ad59-4a68-9dde-80b62492af7f","Type":"ContainerStarted","Data":"9af2a3daac3c3dce93190cea1d5fb808d4468eac657ffa002756ecbcfd08e50d"}
Sep 17 23:12:33.444085 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:33.444062    2584 generic.go:356] "Generic (PLEG): container finished" podID="9d77d57a-58f9-4d62-baef-c4cb8b62816f" containerID="592f81a64b2182caba5a3f845eceb6b2b0f6588fe565ff62729db3a48578ed3d" exitCode=0
Sep 17 23:12:33.444207 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:33.444101    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9d77d57a-58f9-4d62-baef-c4cb8b62816f","Type":"ContainerDied","Data":"592f81a64b2182caba5a3f845eceb6b2b0f6588fe565ff62729db3a48578ed3d"}
Sep 17 23:12:33.464752 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:33.464682    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:12:33.466498 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:33.466468    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 17 23:12:33.483175 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:33.483147    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:12:33.488327 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:33.488296    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-2qip6h26h9dcd\""
Sep 17 23:12:33.488475 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:33.488459    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Sep 17 23:12:33.488547 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:33.488471    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 17 23:12:33.488547 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:33.488528    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 17 23:12:33.489162 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:33.489140    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 17 23:12:33.489462 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:33.489442    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 17 23:12:33.489605 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:33.489464    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 17 23:12:33.489688 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:33.489607    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 17 23:12:33.489688 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:33.489666    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 17 23:12:33.489796 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:33.489544    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 17 23:12:33.489796 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:33.489549    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 17 23:12:33.489796 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:33.489592    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-jcrm8\""
Sep 17 23:12:33.489796 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:33.489487    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 17 23:12:33.490158 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:33.489499    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 17 23:12:33.490674 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:33.490470    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 17 23:12:33.494664 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:33.492742    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-4bbdp" podStartSLOduration=131.421860037 podStartE2EDuration="2m14.492722657s" totalImagesPullingTime="3.07086262s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:10:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:12:33.473660598 +0000 UTC m=+136.839320515" watchObservedRunningTime="2026-09-17 23:12:33.492722657 +0000 UTC m=+136.858382575"
Sep 17 23:12:33.494664 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:33.494302    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 17 23:12:33.495536 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:33.495337    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 17 23:12:33.496839 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:33.496811    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 17 23:12:33.593092 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:33.593058    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2885d33f-5e94-41ee-aff3-fb77ebcce714-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2885d33f-5e94-41ee-aff3-fb77ebcce714\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:12:33.593294 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:33.593098    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/2885d33f-5e94-41ee-aff3-fb77ebcce714-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"2885d33f-5e94-41ee-aff3-fb77ebcce714\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:12:33.593294 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:33.593125    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2885d33f-5e94-41ee-aff3-fb77ebcce714-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"2885d33f-5e94-41ee-aff3-fb77ebcce714\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:12:33.593294 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:33.593147    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/2885d33f-5e94-41ee-aff3-fb77ebcce714-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"2885d33f-5e94-41ee-aff3-fb77ebcce714\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:12:33.593294 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:33.593170    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/2885d33f-5e94-41ee-aff3-fb77ebcce714-web-config\") pod \"prometheus-k8s-0\" (UID: \"2885d33f-5e94-41ee-aff3-fb77ebcce714\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:12:33.593294 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:33.593205    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/2885d33f-5e94-41ee-aff3-fb77ebcce714-config-out\") pod \"prometheus-k8s-0\" (UID: \"2885d33f-5e94-41ee-aff3-fb77ebcce714\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:12:33.593294 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:33.593230    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/2885d33f-5e94-41ee-aff3-fb77ebcce714-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"2885d33f-5e94-41ee-aff3-fb77ebcce714\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:12:33.593483 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:33.593296    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2885d33f-5e94-41ee-aff3-fb77ebcce714-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2885d33f-5e94-41ee-aff3-fb77ebcce714\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:12:33.593483 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:33.593320    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/2885d33f-5e94-41ee-aff3-fb77ebcce714-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"2885d33f-5e94-41ee-aff3-fb77ebcce714\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:12:33.593483 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:33.593339    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2885d33f-5e94-41ee-aff3-fb77ebcce714-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"2885d33f-5e94-41ee-aff3-fb77ebcce714\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:12:33.593483 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:33.593356    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2885d33f-5e94-41ee-aff3-fb77ebcce714-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"2885d33f-5e94-41ee-aff3-fb77ebcce714\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:12:33.593483 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:33.593382    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/2885d33f-5e94-41ee-aff3-fb77ebcce714-config\") pod \"prometheus-k8s-0\" (UID: \"2885d33f-5e94-41ee-aff3-fb77ebcce714\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:12:33.593483 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:33.593404    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/2885d33f-5e94-41ee-aff3-fb77ebcce714-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"2885d33f-5e94-41ee-aff3-fb77ebcce714\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:12:33.593483 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:33.593436    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/2885d33f-5e94-41ee-aff3-fb77ebcce714-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"2885d33f-5e94-41ee-aff3-fb77ebcce714\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:12:33.593483 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:33.593460    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2885d33f-5e94-41ee-aff3-fb77ebcce714-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2885d33f-5e94-41ee-aff3-fb77ebcce714\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:12:33.593483 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:33.593478    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/2885d33f-5e94-41ee-aff3-fb77ebcce714-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"2885d33f-5e94-41ee-aff3-fb77ebcce714\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:12:33.593766 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:33.593491    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qdq27\" (UniqueName: \"kubernetes.io/projected/2885d33f-5e94-41ee-aff3-fb77ebcce714-kube-api-access-qdq27\") pod \"prometheus-k8s-0\" (UID: \"2885d33f-5e94-41ee-aff3-fb77ebcce714\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:12:33.593766 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:33.593544    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/2885d33f-5e94-41ee-aff3-fb77ebcce714-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"2885d33f-5e94-41ee-aff3-fb77ebcce714\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:12:33.593766 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:33.593569    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/2885d33f-5e94-41ee-aff3-fb77ebcce714-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"2885d33f-5e94-41ee-aff3-fb77ebcce714\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:12:33.593766 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:33.593600    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/2885d33f-5e94-41ee-aff3-fb77ebcce714-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"2885d33f-5e94-41ee-aff3-fb77ebcce714\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:12:33.649992 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:33.649945    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:12:33.694135 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:33.694042    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2885d33f-5e94-41ee-aff3-fb77ebcce714-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2885d33f-5e94-41ee-aff3-fb77ebcce714\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:12:33.695322 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:33.695301    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/2885d33f-5e94-41ee-aff3-fb77ebcce714-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"2885d33f-5e94-41ee-aff3-fb77ebcce714\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:12:33.695906 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:33.695869    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qdq27\" (UniqueName: \"kubernetes.io/projected/2885d33f-5e94-41ee-aff3-fb77ebcce714-kube-api-access-qdq27\") pod \"prometheus-k8s-0\" (UID: \"2885d33f-5e94-41ee-aff3-fb77ebcce714\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:12:33.696026 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:33.695231    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2885d33f-5e94-41ee-aff3-fb77ebcce714-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2885d33f-5e94-41ee-aff3-fb77ebcce714\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:12:33.696026 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:33.695905    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/2885d33f-5e94-41ee-aff3-fb77ebcce714-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"2885d33f-5e94-41ee-aff3-fb77ebcce714\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:12:33.696140 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:33.696000    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/2885d33f-5e94-41ee-aff3-fb77ebcce714-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"2885d33f-5e94-41ee-aff3-fb77ebcce714\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:12:33.699124 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:33.696613    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/2885d33f-5e94-41ee-aff3-fb77ebcce714-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"2885d33f-5e94-41ee-aff3-fb77ebcce714\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:12:33.699124 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:33.696722    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2885d33f-5e94-41ee-aff3-fb77ebcce714-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2885d33f-5e94-41ee-aff3-fb77ebcce714\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:12:33.699124 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:33.696857    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/2885d33f-5e94-41ee-aff3-fb77ebcce714-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"2885d33f-5e94-41ee-aff3-fb77ebcce714\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:12:33.699124 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:33.696895    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2885d33f-5e94-41ee-aff3-fb77ebcce714-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"2885d33f-5e94-41ee-aff3-fb77ebcce714\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:12:33.699124 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:33.697139    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/2885d33f-5e94-41ee-aff3-fb77ebcce714-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"2885d33f-5e94-41ee-aff3-fb77ebcce714\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:12:33.699124 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:33.697174    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/2885d33f-5e94-41ee-aff3-fb77ebcce714-web-config\") pod \"prometheus-k8s-0\" (UID: \"2885d33f-5e94-41ee-aff3-fb77ebcce714\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:12:33.699124 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:33.697222    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/2885d33f-5e94-41ee-aff3-fb77ebcce714-config-out\") pod \"prometheus-k8s-0\" (UID: \"2885d33f-5e94-41ee-aff3-fb77ebcce714\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:12:33.699124 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:33.697345    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/2885d33f-5e94-41ee-aff3-fb77ebcce714-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"2885d33f-5e94-41ee-aff3-fb77ebcce714\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:12:33.699124 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:33.697651    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/2885d33f-5e94-41ee-aff3-fb77ebcce714-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"2885d33f-5e94-41ee-aff3-fb77ebcce714\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:12:33.699124 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:33.697982    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2885d33f-5e94-41ee-aff3-fb77ebcce714-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2885d33f-5e94-41ee-aff3-fb77ebcce714\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:12:33.699124 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:33.698878    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/2885d33f-5e94-41ee-aff3-fb77ebcce714-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"2885d33f-5e94-41ee-aff3-fb77ebcce714\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:12:33.699124 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:33.698889    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2885d33f-5e94-41ee-aff3-fb77ebcce714-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"2885d33f-5e94-41ee-aff3-fb77ebcce714\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:12:33.699843 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:33.699395    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2885d33f-5e94-41ee-aff3-fb77ebcce714-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2885d33f-5e94-41ee-aff3-fb77ebcce714\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:12:33.699843 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:33.699483    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/2885d33f-5e94-41ee-aff3-fb77ebcce714-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"2885d33f-5e94-41ee-aff3-fb77ebcce714\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:12:33.699843 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:33.699558    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2885d33f-5e94-41ee-aff3-fb77ebcce714-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"2885d33f-5e94-41ee-aff3-fb77ebcce714\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:12:33.699843 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:33.699600    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2885d33f-5e94-41ee-aff3-fb77ebcce714-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"2885d33f-5e94-41ee-aff3-fb77ebcce714\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:12:33.699843 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:33.699669    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/2885d33f-5e94-41ee-aff3-fb77ebcce714-config\") pod \"prometheus-k8s-0\" (UID: \"2885d33f-5e94-41ee-aff3-fb77ebcce714\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:12:33.699843 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:33.699732    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/2885d33f-5e94-41ee-aff3-fb77ebcce714-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"2885d33f-5e94-41ee-aff3-fb77ebcce714\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:12:33.699843 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:33.699796    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/2885d33f-5e94-41ee-aff3-fb77ebcce714-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"2885d33f-5e94-41ee-aff3-fb77ebcce714\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:12:33.701038 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:33.700951    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/2885d33f-5e94-41ee-aff3-fb77ebcce714-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"2885d33f-5e94-41ee-aff3-fb77ebcce714\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:12:33.701959 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:33.701929    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/2885d33f-5e94-41ee-aff3-fb77ebcce714-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"2885d33f-5e94-41ee-aff3-fb77ebcce714\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:12:33.702337 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:33.702315    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/2885d33f-5e94-41ee-aff3-fb77ebcce714-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"2885d33f-5e94-41ee-aff3-fb77ebcce714\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:12:33.702544 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:33.702525    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/2885d33f-5e94-41ee-aff3-fb77ebcce714-config-out\") pod \"prometheus-k8s-0\" (UID: \"2885d33f-5e94-41ee-aff3-fb77ebcce714\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:12:33.703543 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:33.703454    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2885d33f-5e94-41ee-aff3-fb77ebcce714-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"2885d33f-5e94-41ee-aff3-fb77ebcce714\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:12:33.705332 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:33.704106    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2885d33f-5e94-41ee-aff3-fb77ebcce714-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"2885d33f-5e94-41ee-aff3-fb77ebcce714\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:12:33.705332 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:33.704283    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/2885d33f-5e94-41ee-aff3-fb77ebcce714-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"2885d33f-5e94-41ee-aff3-fb77ebcce714\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:12:33.705332 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:33.704604    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2885d33f-5e94-41ee-aff3-fb77ebcce714-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2885d33f-5e94-41ee-aff3-fb77ebcce714\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:12:33.705332 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:33.705304    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/2885d33f-5e94-41ee-aff3-fb77ebcce714-web-config\") pod \"prometheus-k8s-0\" (UID: \"2885d33f-5e94-41ee-aff3-fb77ebcce714\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:12:33.705896 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:33.705677    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/2885d33f-5e94-41ee-aff3-fb77ebcce714-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"2885d33f-5e94-41ee-aff3-fb77ebcce714\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:12:33.707948 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:33.707697    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/2885d33f-5e94-41ee-aff3-fb77ebcce714-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"2885d33f-5e94-41ee-aff3-fb77ebcce714\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:12:33.709687 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:33.709667    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/2885d33f-5e94-41ee-aff3-fb77ebcce714-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"2885d33f-5e94-41ee-aff3-fb77ebcce714\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:12:33.709793 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:33.709771    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/2885d33f-5e94-41ee-aff3-fb77ebcce714-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"2885d33f-5e94-41ee-aff3-fb77ebcce714\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:12:33.709992 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:33.709971    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/2885d33f-5e94-41ee-aff3-fb77ebcce714-config\") pod \"prometheus-k8s-0\" (UID: \"2885d33f-5e94-41ee-aff3-fb77ebcce714\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:12:33.710728 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:33.710708    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qdq27\" (UniqueName: \"kubernetes.io/projected/2885d33f-5e94-41ee-aff3-fb77ebcce714-kube-api-access-qdq27\") pod \"prometheus-k8s-0\" (UID: \"2885d33f-5e94-41ee-aff3-fb77ebcce714\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:12:33.751667 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:33.751621    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:12:33.797387 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:33.797352    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:12:34.128665 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:34.128629    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 17 23:12:34.132418 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:12:34.132387    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2885d33f_5e94_41ee_aff3_fb77ebcce714.slice/crio-5af7b1687c08549a51b04a634b0d35331346f2454209a5497388eeee2224b384 WatchSource:0}: Error finding container 5af7b1687c08549a51b04a634b0d35331346f2454209a5497388eeee2224b384: Status 404 returned error can't find the container with id 5af7b1687c08549a51b04a634b0d35331346f2454209a5497388eeee2224b384
Sep 17 23:12:34.270203 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:34.270171    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:12:34.363315 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:34.363262    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:12:34.450266 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:34.450187    2584 generic.go:356] "Generic (PLEG): container finished" podID="2885d33f-5e94-41ee-aff3-fb77ebcce714" containerID="0d1fc61add4612ec08b45e6b14835e849fb73e4aa08c62833e1c1f1920a7a7d9" exitCode=0
Sep 17 23:12:34.450739 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:34.450285    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2885d33f-5e94-41ee-aff3-fb77ebcce714","Type":"ContainerDied","Data":"0d1fc61add4612ec08b45e6b14835e849fb73e4aa08c62833e1c1f1920a7a7d9"}
Sep 17 23:12:34.450739 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:34.450366    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2885d33f-5e94-41ee-aff3-fb77ebcce714","Type":"ContainerStarted","Data":"5af7b1687c08549a51b04a634b0d35331346f2454209a5497388eeee2224b384"}
Sep 17 23:12:35.456694 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:35.456658    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9d77d57a-58f9-4d62-baef-c4cb8b62816f","Type":"ContainerStarted","Data":"9bc2ff4010dd58a6450c970af8fe887881e659ddddd53c66335555935221079f"}
Sep 17 23:12:36.038597 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:36.038563    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:12:36.144606 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:36.144572    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:12:36.467028 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:36.466969    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6c99487fbd-897xp" event={"ID":"e2b185f4-e371-4556-a3ae-3eda3d226876","Type":"ContainerStarted","Data":"0b2731a2b0745054a091bb0a0498874e69f4d6662beff52c049731d0b94aa928"}
Sep 17 23:12:36.467493 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:36.467111    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6c99487fbd-897xp" event={"ID":"e2b185f4-e371-4556-a3ae-3eda3d226876","Type":"ContainerStarted","Data":"33121dbe1b840070a5bd2ae192eef46141682dc59bd45d4cea860fb087e38287"}
Sep 17 23:12:36.467493 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:36.467155    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6c99487fbd-897xp" event={"ID":"e2b185f4-e371-4556-a3ae-3eda3d226876","Type":"ContainerStarted","Data":"1ed1254145811c1a7a9ebb6e12fff1d049c4b2a89b48dff29af31dbd3745de75"}
Sep 17 23:12:36.467493 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:36.467296    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-6c99487fbd-897xp"
Sep 17 23:12:36.471308 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:36.471262    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9d77d57a-58f9-4d62-baef-c4cb8b62816f","Type":"ContainerStarted","Data":"30f7391271d53334fb7aa3504ba9df2f6ced882683e96ecbacadefb65cfc1498"}
Sep 17 23:12:36.471462 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:36.471338    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9d77d57a-58f9-4d62-baef-c4cb8b62816f","Type":"ContainerStarted","Data":"ccd4c3c365c10c690dec0564b494d275cecedd0db1dccb2529f877c470abcee0"}
Sep 17 23:12:36.471462 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:36.471378    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9d77d57a-58f9-4d62-baef-c4cb8b62816f","Type":"ContainerStarted","Data":"5817c97f2ee92091c3bc1e56e832110f7e9075f288a8a71a68977ddc1e4331d6"}
Sep 17 23:12:36.471462 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:36.471416    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9d77d57a-58f9-4d62-baef-c4cb8b62816f","Type":"ContainerStarted","Data":"d300fe349a2c8408220010b09ebbd0dc440ca229535d12af03494d283814f52c"}
Sep 17 23:12:36.471462 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:36.471453    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9d77d57a-58f9-4d62-baef-c4cb8b62816f","Type":"ContainerStarted","Data":"26f29b93082118a8cb8ab07e4c49c46bcbdc926e2fcc157c9996dd3e8b1aa835"}
Sep 17 23:12:36.567145 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:36.567029    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-6c99487fbd-897xp" podStartSLOduration=2.709026692 podStartE2EDuration="8.566986746s" totalImagesPullingTime="5.857960054s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:12:28 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:12:36.563979698 +0000 UTC m=+139.929639616" watchObservedRunningTime="2026-09-17 23:12:36.566986746 +0000 UTC m=+139.932646663"
Sep 17 23:12:36.629227 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:36.629160    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=4.784141125 podStartE2EDuration="9.629134728s" totalImagesPullingTime="4.830096576s" totalInitContainerRuntime="14.897027ms" isStatefulPod=false podCreationTimestamp="2026-09-17 23:12:27 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:12:36.624586561 +0000 UTC m=+139.990246490" watchObservedRunningTime="2026-09-17 23:12:36.629134728 +0000 UTC m=+139.994794649"
Sep 17 23:12:38.179855 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:38.179823    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:12:38.279057 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:38.279023    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:12:39.488897 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:39.488864    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2885d33f-5e94-41ee-aff3-fb77ebcce714","Type":"ContainerStarted","Data":"3e90483fc433aae8c75161200ebb17bded81d1ab2018e2eb27fba3c8800f3999"}
Sep 17 23:12:39.489375 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:39.488976    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2885d33f-5e94-41ee-aff3-fb77ebcce714","Type":"ContainerStarted","Data":"44a2e8fdaca7a912cc9d49c9e5ab6e285282e40cd939fe6662a20b400b1f4440"}
Sep 17 23:12:39.489375 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:39.489049    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2885d33f-5e94-41ee-aff3-fb77ebcce714","Type":"ContainerStarted","Data":"16565b79796f570b0010257d217782384e51e052a65a9e01ad7dcbc7ac244e45"}
Sep 17 23:12:39.489375 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:39.489106    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2885d33f-5e94-41ee-aff3-fb77ebcce714","Type":"ContainerStarted","Data":"7c1de2e9266451f75c0267803a6bd913837c8e674dddb3a81b2958319b776a3f"}
Sep 17 23:12:39.489375 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:39.489146    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2885d33f-5e94-41ee-aff3-fb77ebcce714","Type":"ContainerStarted","Data":"65b09b45b64996336e719015fc4c618c0065448263e1e1f0f1bd18763fd38a53"}
Sep 17 23:12:39.489375 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:39.489188    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2885d33f-5e94-41ee-aff3-fb77ebcce714","Type":"ContainerStarted","Data":"499aff5f879614ed1f1b508544f1bec6e3defc79a4d6eb79e5754f0abf91ff19"}
Sep 17 23:12:39.539928 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:39.538146    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.413949804 podStartE2EDuration="6.538124162s" totalImagesPullingTime="4.094691337s" totalInitContainerRuntime="29.483021ms" isStatefulPod=false podCreationTimestamp="2026-09-17 23:12:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:12:39.534835668 +0000 UTC m=+142.900495584" watchObservedRunningTime="2026-09-17 23:12:39.538124162 +0000 UTC m=+142.903784080"
Sep 17 23:12:39.668534 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:39.668446    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:12:39.780957 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:39.780919    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:12:39.883329 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:39.883303    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:12:40.378539 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:40.378506    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:12:40.488605 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:40.488572    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:12:41.072132 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:41.072093    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:12:41.179644 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:41.179617    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:12:42.481897 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:42.481861    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-6c99487fbd-897xp"
Sep 17 23:12:43.797634 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:43.797604    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:12:44.617648 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:44.617609    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:12:44.743230 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:44.743175    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:12:46.545890 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:46.545854    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:12:46.654960 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:46.654930    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:12:48.860500 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:48.860431    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-75f6fdbdff-5rgvh" podUID="06535038-f186-42c3-b9d1-707e2c7ea285" containerName="registry" containerID="cri-o://f1f191b3b0e62d08218cb5c0b74f2abd552d690950d7a5e0c9010f83ed0d22e0" gracePeriod=30
Sep 17 23:12:49.122434 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:49.122361    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-75f6fdbdff-5rgvh"
Sep 17 23:12:49.245867 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:49.245832    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/06535038-f186-42c3-b9d1-707e2c7ea285-bound-sa-token\") pod \"06535038-f186-42c3-b9d1-707e2c7ea285\" (UID: \"06535038-f186-42c3-b9d1-707e2c7ea285\") "
Sep 17 23:12:49.246094 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:49.245874    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/06535038-f186-42c3-b9d1-707e2c7ea285-image-registry-private-configuration\") pod \"06535038-f186-42c3-b9d1-707e2c7ea285\" (UID: \"06535038-f186-42c3-b9d1-707e2c7ea285\") "
Sep 17 23:12:49.246094 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:49.245915    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/06535038-f186-42c3-b9d1-707e2c7ea285-registry-certificates\") pod \"06535038-f186-42c3-b9d1-707e2c7ea285\" (UID: \"06535038-f186-42c3-b9d1-707e2c7ea285\") "
Sep 17 23:12:49.246094 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:49.245937    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/06535038-f186-42c3-b9d1-707e2c7ea285-installation-pull-secrets\") pod \"06535038-f186-42c3-b9d1-707e2c7ea285\" (UID: \"06535038-f186-42c3-b9d1-707e2c7ea285\") "
Sep 17 23:12:49.246094 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:49.245977    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dj65p\" (UniqueName: \"kubernetes.io/projected/06535038-f186-42c3-b9d1-707e2c7ea285-kube-api-access-dj65p\") pod \"06535038-f186-42c3-b9d1-707e2c7ea285\" (UID: \"06535038-f186-42c3-b9d1-707e2c7ea285\") "
Sep 17 23:12:49.246094 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:49.246048    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/06535038-f186-42c3-b9d1-707e2c7ea285-trusted-ca\") pod \"06535038-f186-42c3-b9d1-707e2c7ea285\" (UID: \"06535038-f186-42c3-b9d1-707e2c7ea285\") "
Sep 17 23:12:49.246725 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:49.246087    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/06535038-f186-42c3-b9d1-707e2c7ea285-ca-trust-extracted\") pod \"06535038-f186-42c3-b9d1-707e2c7ea285\" (UID: \"06535038-f186-42c3-b9d1-707e2c7ea285\") "
Sep 17 23:12:49.246725 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:49.246119    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/06535038-f186-42c3-b9d1-707e2c7ea285-registry-tls\") pod \"06535038-f186-42c3-b9d1-707e2c7ea285\" (UID: \"06535038-f186-42c3-b9d1-707e2c7ea285\") "
Sep 17 23:12:49.247050 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:49.246769    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06535038-f186-42c3-b9d1-707e2c7ea285-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "06535038-f186-42c3-b9d1-707e2c7ea285" (UID: "06535038-f186-42c3-b9d1-707e2c7ea285"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:12:49.247050 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:49.246326    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06535038-f186-42c3-b9d1-707e2c7ea285-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "06535038-f186-42c3-b9d1-707e2c7ea285" (UID: "06535038-f186-42c3-b9d1-707e2c7ea285"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:12:49.248847 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:49.248781    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/06535038-f186-42c3-b9d1-707e2c7ea285-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "06535038-f186-42c3-b9d1-707e2c7ea285" (UID: "06535038-f186-42c3-b9d1-707e2c7ea285"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:12:49.248959 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:49.248869    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06535038-f186-42c3-b9d1-707e2c7ea285-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "06535038-f186-42c3-b9d1-707e2c7ea285" (UID: "06535038-f186-42c3-b9d1-707e2c7ea285"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:12:49.248959 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:49.248932    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06535038-f186-42c3-b9d1-707e2c7ea285-kube-api-access-dj65p" (OuterVolumeSpecName: "kube-api-access-dj65p") pod "06535038-f186-42c3-b9d1-707e2c7ea285" (UID: "06535038-f186-42c3-b9d1-707e2c7ea285"). InnerVolumeSpecName "kube-api-access-dj65p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:12:49.248959 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:49.248945    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06535038-f186-42c3-b9d1-707e2c7ea285-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "06535038-f186-42c3-b9d1-707e2c7ea285" (UID: "06535038-f186-42c3-b9d1-707e2c7ea285"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:12:49.249145 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:49.248959    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/06535038-f186-42c3-b9d1-707e2c7ea285-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "06535038-f186-42c3-b9d1-707e2c7ea285" (UID: "06535038-f186-42c3-b9d1-707e2c7ea285"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:12:49.255171 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:49.255148    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06535038-f186-42c3-b9d1-707e2c7ea285-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "06535038-f186-42c3-b9d1-707e2c7ea285" (UID: "06535038-f186-42c3-b9d1-707e2c7ea285"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:12:49.347243 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:49.347209    2584 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/06535038-f186-42c3-b9d1-707e2c7ea285-bound-sa-token\") on node \"ip-10-0-142-4.ec2.internal\" DevicePath \"\""
Sep 17 23:12:49.347243 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:49.347240    2584 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/06535038-f186-42c3-b9d1-707e2c7ea285-image-registry-private-configuration\") on node \"ip-10-0-142-4.ec2.internal\" DevicePath \"\""
Sep 17 23:12:49.347243 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:49.347250    2584 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/06535038-f186-42c3-b9d1-707e2c7ea285-registry-certificates\") on node \"ip-10-0-142-4.ec2.internal\" DevicePath \"\""
Sep 17 23:12:49.347465 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:49.347261    2584 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/06535038-f186-42c3-b9d1-707e2c7ea285-installation-pull-secrets\") on node \"ip-10-0-142-4.ec2.internal\" DevicePath \"\""
Sep 17 23:12:49.347465 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:49.347269    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dj65p\" (UniqueName: \"kubernetes.io/projected/06535038-f186-42c3-b9d1-707e2c7ea285-kube-api-access-dj65p\") on node \"ip-10-0-142-4.ec2.internal\" DevicePath \"\""
Sep 17 23:12:49.347465 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:49.347277    2584 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/06535038-f186-42c3-b9d1-707e2c7ea285-trusted-ca\") on node \"ip-10-0-142-4.ec2.internal\" DevicePath \"\""
Sep 17 23:12:49.347465 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:49.347285    2584 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/06535038-f186-42c3-b9d1-707e2c7ea285-ca-trust-extracted\") on node \"ip-10-0-142-4.ec2.internal\" DevicePath \"\""
Sep 17 23:12:49.347465 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:49.347292    2584 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/06535038-f186-42c3-b9d1-707e2c7ea285-registry-tls\") on node \"ip-10-0-142-4.ec2.internal\" DevicePath \"\""
Sep 17 23:12:49.532973 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:49.532936    2584 generic.go:356] "Generic (PLEG): container finished" podID="06535038-f186-42c3-b9d1-707e2c7ea285" containerID="f1f191b3b0e62d08218cb5c0b74f2abd552d690950d7a5e0c9010f83ed0d22e0" exitCode=0
Sep 17 23:12:49.533222 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:49.532980    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-75f6fdbdff-5rgvh"
Sep 17 23:12:49.533222 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:49.533027    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-75f6fdbdff-5rgvh" event={"ID":"06535038-f186-42c3-b9d1-707e2c7ea285","Type":"ContainerDied","Data":"f1f191b3b0e62d08218cb5c0b74f2abd552d690950d7a5e0c9010f83ed0d22e0"}
Sep 17 23:12:49.533222 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:49.533080    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-75f6fdbdff-5rgvh" event={"ID":"06535038-f186-42c3-b9d1-707e2c7ea285","Type":"ContainerDied","Data":"9c412698d32fe9f3c7257914d99b86c225846f63d9f212d26bd475c962971aa9"}
Sep 17 23:12:49.533222 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:49.533112    2584 scope.go:117] "RemoveContainer" containerID="f1f191b3b0e62d08218cb5c0b74f2abd552d690950d7a5e0c9010f83ed0d22e0"
Sep 17 23:12:49.542346 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:49.542326    2584 scope.go:117] "RemoveContainer" containerID="f1f191b3b0e62d08218cb5c0b74f2abd552d690950d7a5e0c9010f83ed0d22e0"
Sep 17 23:12:49.542619 ip-10-0-142-4 kubenswrapper[2584]: E0917 23:12:49.542598    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f1f191b3b0e62d08218cb5c0b74f2abd552d690950d7a5e0c9010f83ed0d22e0\": container with ID starting with f1f191b3b0e62d08218cb5c0b74f2abd552d690950d7a5e0c9010f83ed0d22e0 not found: ID does not exist" containerID="f1f191b3b0e62d08218cb5c0b74f2abd552d690950d7a5e0c9010f83ed0d22e0"
Sep 17 23:12:49.542690 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:49.542631    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1f191b3b0e62d08218cb5c0b74f2abd552d690950d7a5e0c9010f83ed0d22e0"} err="failed to get container status \"f1f191b3b0e62d08218cb5c0b74f2abd552d690950d7a5e0c9010f83ed0d22e0\": rpc error: code = NotFound desc = could not find container \"f1f191b3b0e62d08218cb5c0b74f2abd552d690950d7a5e0c9010f83ed0d22e0\": container with ID starting with f1f191b3b0e62d08218cb5c0b74f2abd552d690950d7a5e0c9010f83ed0d22e0 not found: ID does not exist"
Sep 17 23:12:49.564142 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:49.564118    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-75f6fdbdff-5rgvh"]
Sep 17 23:12:49.577952 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:49.577927    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-75f6fdbdff-5rgvh"]
Sep 17 23:12:50.765199 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:50.765170    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="06535038-f186-42c3-b9d1-707e2c7ea285" path="/var/lib/kubelet/pods/06535038-f186-42c3-b9d1-707e2c7ea285/volumes"
Sep 17 23:12:59.684842 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:59.684708    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:12:59.923523 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:12:59.923486    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:13:33.797556 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:13:33.797513    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:13:33.820117 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:13:33.820088    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:13:34.733548 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:13:34.733522    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:13:46.573448 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:13:46.573409    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:13:46.676722 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:13:46.676665    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:13:55.739338 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:13:55.739282    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:13:55.840159 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:13:55.840124    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:14:01.928144 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:14:01.928105    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:14:02.027715 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:14:02.027683    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:14:13.794907 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:14:13.794872    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:14:13.895239 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:14:13.895206    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:14:13.980019 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:14:13.979969    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:14:16.053849 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:14:16.053820    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:14:16.134458 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:14:16.134431    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:14:17.265436 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:14:17.265403    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:14:17.400307 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:14:17.400273    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:14:17.530071 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:14:17.529968    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:14:37.118343 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:14:37.118269    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:14:56.013517 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:14:56.013488    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:14:56.097415 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:14:56.097381    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:15:18.683200 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:15:18.683162    2584 kubelet.go:1648] "Image garbage collection succeeded"
Sep 17 23:15:19.232085 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:15:19.232051    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:15:19.379784 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:15:19.379758    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:15:24.626434 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:15:24.626400    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:15:24.699558 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:15:24.699527    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:15:25.504302 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:15:25.504272    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:15:25.791572 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:15:25.791489    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:15:26.039984 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:15:26.039959    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:15:26.142922 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:15:26.142852    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:15:26.239709 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:15:26.239679    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:15:26.939855 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:15:26.939826    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:15:27.130157 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:15:27.130126    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:15:27.234064 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:15:27.233965    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:15:27.580942 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:15:27.580908    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:15:27.697137 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:15:27.697112    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:15:27.827881 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:15:27.827851    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:15:28.045957 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:15:28.045931    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:15:28.227298 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:15:28.227266    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:15:33.947617 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:15:33.947585    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:16:17.556247 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:16:17.556166    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:16:18.923440 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:16:18.923406    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:16:49.902273 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:16:49.902231    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:16:50.021466 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:16:50.021436    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:16:50.148993 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:16:50.148961    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:16:50.258096 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:16:50.258070    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:16:50.382663 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:16:50.382637    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:16:50.562779 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:16:50.562717    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:16:50.928143 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:16:50.928064    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:16:51.018163 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:16:51.018138    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:16:51.899283 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:16:51.899252    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:17:03.436029 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:17:03.435977    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:17:03.535608 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:17:03.535578    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:17:06.249721 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:17:06.249686    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:17:06.330198 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:17:06.330166    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:17:08.326331 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:17:08.326300    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:17:08.507453 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:17:08.507426    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:17:19.599640 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:17:19.599600    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:17:19.699017 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:17:19.698974    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:17:19.778051 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:17:19.778017    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:17:24.854211 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:17:24.854182    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:17:24.915889 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:17:24.915860    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:17:29.969823 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:17:29.969731    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:18:07.824654 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:18:07.824616    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:18:07.931510 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:18:07.931472    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:18:08.035263 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:18:08.035228    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:18:08.145853 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:18:08.145768    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:18:09.331716 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:18:09.331680    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:18:09.403596 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:18:09.403567    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:18:24.962168 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:18:24.962135    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-init-rhtfb"]
Sep 17 23:18:24.962532 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:18:24.962441    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="06535038-f186-42c3-b9d1-707e2c7ea285" containerName="registry"
Sep 17 23:18:24.962532 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:18:24.962452    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="06535038-f186-42c3-b9d1-707e2c7ea285" containerName="registry"
Sep 17 23:18:24.962532 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:18:24.962515    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="06535038-f186-42c3-b9d1-707e2c7ea285" containerName="registry"
Sep 17 23:18:24.964371 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:18:24.964353    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-rhtfb"
Sep 17 23:18:24.966904 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:18:24.966884    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"hf-token\""
Sep 17 23:18:24.967050 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:18:24.966885    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-fmvmx\""
Sep 17 23:18:24.967168 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:18:24.967154    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Sep 17 23:18:24.967744 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:18:24.967728    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Sep 17 23:18:24.967839 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:18:24.967824    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Sep 17 23:18:24.973250 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:18:24.973230    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-rhtfb"]
Sep 17 23:18:25.016341 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:18:25.016306    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e4e101e1-5b0b-4b40-99cd-3c95101e61d4-model-cache\") pod \"s3-init-rhtfb\" (UID: \"e4e101e1-5b0b-4b40-99cd-3c95101e61d4\") " pod="kserve/s3-init-rhtfb"
Sep 17 23:18:25.016518 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:18:25.016387    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5sjp5\" (UniqueName: \"kubernetes.io/projected/e4e101e1-5b0b-4b40-99cd-3c95101e61d4-kube-api-access-5sjp5\") pod \"s3-init-rhtfb\" (UID: \"e4e101e1-5b0b-4b40-99cd-3c95101e61d4\") " pod="kserve/s3-init-rhtfb"
Sep 17 23:18:25.117057 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:18:25.116997    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5sjp5\" (UniqueName: \"kubernetes.io/projected/e4e101e1-5b0b-4b40-99cd-3c95101e61d4-kube-api-access-5sjp5\") pod \"s3-init-rhtfb\" (UID: \"e4e101e1-5b0b-4b40-99cd-3c95101e61d4\") " pod="kserve/s3-init-rhtfb"
Sep 17 23:18:25.117237 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:18:25.117074    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e4e101e1-5b0b-4b40-99cd-3c95101e61d4-model-cache\") pod \"s3-init-rhtfb\" (UID: \"e4e101e1-5b0b-4b40-99cd-3c95101e61d4\") " pod="kserve/s3-init-rhtfb"
Sep 17 23:18:25.117430 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:18:25.117414    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e4e101e1-5b0b-4b40-99cd-3c95101e61d4-model-cache\") pod \"s3-init-rhtfb\" (UID: \"e4e101e1-5b0b-4b40-99cd-3c95101e61d4\") " pod="kserve/s3-init-rhtfb"
Sep 17 23:18:25.125887 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:18:25.125865    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5sjp5\" (UniqueName: \"kubernetes.io/projected/e4e101e1-5b0b-4b40-99cd-3c95101e61d4-kube-api-access-5sjp5\") pod \"s3-init-rhtfb\" (UID: \"e4e101e1-5b0b-4b40-99cd-3c95101e61d4\") " pod="kserve/s3-init-rhtfb"
Sep 17 23:18:25.280951 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:18:25.280919    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-rhtfb"
Sep 17 23:18:25.556235 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:18:25.556212    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-rhtfb"]
Sep 17 23:18:25.558380 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:18:25.558356    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode4e101e1_5b0b_4b40_99cd_3c95101e61d4.slice/crio-9bdc9bf2211b88451fde622be95c83700ceb6c787a57937465882180cd2d3d25 WatchSource:0}: Error finding container 9bdc9bf2211b88451fde622be95c83700ceb6c787a57937465882180cd2d3d25: Status 404 returned error can't find the container with id 9bdc9bf2211b88451fde622be95c83700ceb6c787a57937465882180cd2d3d25
Sep 17 23:18:25.560268 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:18:25.560253    2584 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 23:18:25.814278 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:18:25.814192    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-rhtfb" event={"ID":"e4e101e1-5b0b-4b40-99cd-3c95101e61d4","Type":"ContainerStarted","Data":"9bdc9bf2211b88451fde622be95c83700ceb6c787a57937465882180cd2d3d25"}
Sep 17 23:18:29.834858 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:18:29.834817    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-rhtfb" event={"ID":"e4e101e1-5b0b-4b40-99cd-3c95101e61d4","Type":"ContainerStarted","Data":"30abacf036b9ea399aff472a67a4bef2c263d56b94fa3021bb89e314cef6f3f4"}
Sep 17 23:18:31.846093 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:18:31.846053    2584 generic.go:356] "Generic (PLEG): container finished" podID="e4e101e1-5b0b-4b40-99cd-3c95101e61d4" containerID="30abacf036b9ea399aff472a67a4bef2c263d56b94fa3021bb89e314cef6f3f4" exitCode=0
Sep 17 23:18:31.846584 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:18:31.846100    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-rhtfb" event={"ID":"e4e101e1-5b0b-4b40-99cd-3c95101e61d4","Type":"ContainerDied","Data":"30abacf036b9ea399aff472a67a4bef2c263d56b94fa3021bb89e314cef6f3f4"}
Sep 17 23:18:36.872933 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:18:36.872895    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-rhtfb" event={"ID":"e4e101e1-5b0b-4b40-99cd-3c95101e61d4","Type":"ContainerStarted","Data":"5c2e982d58b0b62335012aa7fc2405686da0d4338dcfb4d08e1c92bc5c81be60"}
Sep 17 23:19:42.122505 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:19:42.122471    2584 generic.go:356] "Generic (PLEG): container finished" podID="e4e101e1-5b0b-4b40-99cd-3c95101e61d4" containerID="5c2e982d58b0b62335012aa7fc2405686da0d4338dcfb4d08e1c92bc5c81be60" exitCode=0
Sep 17 23:19:42.122892 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:19:42.122518    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-rhtfb" event={"ID":"e4e101e1-5b0b-4b40-99cd-3c95101e61d4","Type":"ContainerDied","Data":"5c2e982d58b0b62335012aa7fc2405686da0d4338dcfb4d08e1c92bc5c81be60"}
Sep 17 23:19:43.250233 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:19:43.250207    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-rhtfb"
Sep 17 23:19:43.337140 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:19:43.337112    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5sjp5\" (UniqueName: \"kubernetes.io/projected/e4e101e1-5b0b-4b40-99cd-3c95101e61d4-kube-api-access-5sjp5\") pod \"e4e101e1-5b0b-4b40-99cd-3c95101e61d4\" (UID: \"e4e101e1-5b0b-4b40-99cd-3c95101e61d4\") "
Sep 17 23:19:43.337140 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:19:43.337143    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e4e101e1-5b0b-4b40-99cd-3c95101e61d4-model-cache\") pod \"e4e101e1-5b0b-4b40-99cd-3c95101e61d4\" (UID: \"e4e101e1-5b0b-4b40-99cd-3c95101e61d4\") "
Sep 17 23:19:43.339322 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:19:43.339285    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4e101e1-5b0b-4b40-99cd-3c95101e61d4-kube-api-access-5sjp5" (OuterVolumeSpecName: "kube-api-access-5sjp5") pod "e4e101e1-5b0b-4b40-99cd-3c95101e61d4" (UID: "e4e101e1-5b0b-4b40-99cd-3c95101e61d4"). InnerVolumeSpecName "kube-api-access-5sjp5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:19:43.355796 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:19:43.355773    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4e101e1-5b0b-4b40-99cd-3c95101e61d4-model-cache" (OuterVolumeSpecName: "model-cache") pod "e4e101e1-5b0b-4b40-99cd-3c95101e61d4" (UID: "e4e101e1-5b0b-4b40-99cd-3c95101e61d4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:19:43.438050 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:19:43.437971    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5sjp5\" (UniqueName: \"kubernetes.io/projected/e4e101e1-5b0b-4b40-99cd-3c95101e61d4-kube-api-access-5sjp5\") on node \"ip-10-0-142-4.ec2.internal\" DevicePath \"\""
Sep 17 23:19:43.438050 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:19:43.437993    2584 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e4e101e1-5b0b-4b40-99cd-3c95101e61d4-model-cache\") on node \"ip-10-0-142-4.ec2.internal\" DevicePath \"\""
Sep 17 23:19:44.131406 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:19:44.131378    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-rhtfb"
Sep 17 23:19:44.131566 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:19:44.131413    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-rhtfb" event={"ID":"e4e101e1-5b0b-4b40-99cd-3c95101e61d4","Type":"ContainerDied","Data":"9bdc9bf2211b88451fde622be95c83700ceb6c787a57937465882180cd2d3d25"}
Sep 17 23:19:44.131566 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:19:44.131451    2584 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9bdc9bf2211b88451fde622be95c83700ceb6c787a57937465882180cd2d3d25"
Sep 17 23:19:44.566549 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:19:44.566516    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:19:44.651831 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:19:44.651801    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:19:56.055982 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:19:56.055931    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:19:56.120827 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:19:56.120798    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a264e-predictor-7bc5f796dc-gxcgj"]
Sep 17 23:19:56.121122 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:19:56.121109    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e4e101e1-5b0b-4b40-99cd-3c95101e61d4" containerName="download-hf-model"
Sep 17 23:19:56.121173 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:19:56.121124    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="e4e101e1-5b0b-4b40-99cd-3c95101e61d4" containerName="download-hf-model"
Sep 17 23:19:56.121173 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:19:56.121140    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e4e101e1-5b0b-4b40-99cd-3c95101e61d4" containerName="s3-init"
Sep 17 23:19:56.121173 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:19:56.121145    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="e4e101e1-5b0b-4b40-99cd-3c95101e61d4" containerName="s3-init"
Sep 17 23:19:56.121272 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:19:56.121205    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="e4e101e1-5b0b-4b40-99cd-3c95101e61d4" containerName="s3-init"
Sep 17 23:19:56.123542 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:19:56.123524    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a264e-predictor-7bc5f796dc-gxcgj"
Sep 17 23:19:56.125945 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:19:56.125913    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-raw-sklearn-batcher-a264e-predictor-serving-cert\""
Sep 17 23:19:56.126083 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:19:56.126046    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-f2cbg\""
Sep 17 23:19:56.126083 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:19:56.126064    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-raw-sklearn-batcher-a264e-kube-rbac-proxy-sar-config\""
Sep 17 23:19:56.126197 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:19:56.126078    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 23:19:56.126197 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:19:56.126130    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 23:19:56.126410 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:19:56.126394    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 23:19:56.126730 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:19:56.126715    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 23:19:56.135087 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:19:56.135067    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a264e-predictor-7bc5f796dc-gxcgj"]
Sep 17 23:19:56.226028 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:19:56.225990    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b2d4dbd6-6084-46c1-a832-3f09ed3ab603-cabundle-cert\") pod \"isvc-raw-sklearn-batcher-a264e-predictor-7bc5f796dc-gxcgj\" (UID: \"b2d4dbd6-6084-46c1-a832-3f09ed3ab603\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a264e-predictor-7bc5f796dc-gxcgj"
Sep 17 23:19:56.226138 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:19:56.226054    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b2d4dbd6-6084-46c1-a832-3f09ed3ab603-kube-rbac-proxy-sar-config\") pod \"isvc-raw-sklearn-batcher-a264e-predictor-7bc5f796dc-gxcgj\" (UID: \"b2d4dbd6-6084-46c1-a832-3f09ed3ab603\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a264e-predictor-7bc5f796dc-gxcgj"
Sep 17 23:19:56.226138 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:19:56.226105    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b2d4dbd6-6084-46c1-a832-3f09ed3ab603-proxy-tls\") pod \"isvc-raw-sklearn-batcher-a264e-predictor-7bc5f796dc-gxcgj\" (UID: \"b2d4dbd6-6084-46c1-a832-3f09ed3ab603\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a264e-predictor-7bc5f796dc-gxcgj"
Sep 17 23:19:56.226138 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:19:56.226123    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-khjgc\" (UniqueName: \"kubernetes.io/projected/b2d4dbd6-6084-46c1-a832-3f09ed3ab603-kube-api-access-khjgc\") pod \"isvc-raw-sklearn-batcher-a264e-predictor-7bc5f796dc-gxcgj\" (UID: \"b2d4dbd6-6084-46c1-a832-3f09ed3ab603\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a264e-predictor-7bc5f796dc-gxcgj"
Sep 17 23:19:56.226238 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:19:56.226148    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b2d4dbd6-6084-46c1-a832-3f09ed3ab603-kserve-provision-location\") pod \"isvc-raw-sklearn-batcher-a264e-predictor-7bc5f796dc-gxcgj\" (UID: \"b2d4dbd6-6084-46c1-a832-3f09ed3ab603\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a264e-predictor-7bc5f796dc-gxcgj"
Sep 17 23:19:56.255161 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:19:56.255137    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:19:56.326520 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:19:56.326453    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df6589c46-cx4bd"]
Sep 17 23:19:56.326715 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:19:56.326695    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b2d4dbd6-6084-46c1-a832-3f09ed3ab603-cabundle-cert\") pod \"isvc-raw-sklearn-batcher-a264e-predictor-7bc5f796dc-gxcgj\" (UID: \"b2d4dbd6-6084-46c1-a832-3f09ed3ab603\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a264e-predictor-7bc5f796dc-gxcgj"
Sep 17 23:19:56.326766 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:19:56.326735    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b2d4dbd6-6084-46c1-a832-3f09ed3ab603-kube-rbac-proxy-sar-config\") pod \"isvc-raw-sklearn-batcher-a264e-predictor-7bc5f796dc-gxcgj\" (UID: \"b2d4dbd6-6084-46c1-a832-3f09ed3ab603\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a264e-predictor-7bc5f796dc-gxcgj"
Sep 17 23:19:56.326802 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:19:56.326768    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b2d4dbd6-6084-46c1-a832-3f09ed3ab603-proxy-tls\") pod \"isvc-raw-sklearn-batcher-a264e-predictor-7bc5f796dc-gxcgj\" (UID: \"b2d4dbd6-6084-46c1-a832-3f09ed3ab603\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a264e-predictor-7bc5f796dc-gxcgj"
Sep 17 23:19:56.326802 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:19:56.326784    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khjgc\" (UniqueName: \"kubernetes.io/projected/b2d4dbd6-6084-46c1-a832-3f09ed3ab603-kube-api-access-khjgc\") pod \"isvc-raw-sklearn-batcher-a264e-predictor-7bc5f796dc-gxcgj\" (UID: \"b2d4dbd6-6084-46c1-a832-3f09ed3ab603\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a264e-predictor-7bc5f796dc-gxcgj"
Sep 17 23:19:56.326905 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:19:56.326805    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b2d4dbd6-6084-46c1-a832-3f09ed3ab603-kserve-provision-location\") pod \"isvc-raw-sklearn-batcher-a264e-predictor-7bc5f796dc-gxcgj\" (UID: \"b2d4dbd6-6084-46c1-a832-3f09ed3ab603\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a264e-predictor-7bc5f796dc-gxcgj"
Sep 17 23:19:56.327341 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:19:56.327314    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b2d4dbd6-6084-46c1-a832-3f09ed3ab603-kserve-provision-location\") pod \"isvc-raw-sklearn-batcher-a264e-predictor-7bc5f796dc-gxcgj\" (UID: \"b2d4dbd6-6084-46c1-a832-3f09ed3ab603\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a264e-predictor-7bc5f796dc-gxcgj"
Sep 17 23:19:56.327529 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:19:56.327506    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b2d4dbd6-6084-46c1-a832-3f09ed3ab603-kube-rbac-proxy-sar-config\") pod \"isvc-raw-sklearn-batcher-a264e-predictor-7bc5f796dc-gxcgj\" (UID: \"b2d4dbd6-6084-46c1-a832-3f09ed3ab603\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a264e-predictor-7bc5f796dc-gxcgj"
Sep 17 23:19:56.327529 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:19:56.327522    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b2d4dbd6-6084-46c1-a832-3f09ed3ab603-cabundle-cert\") pod \"isvc-raw-sklearn-batcher-a264e-predictor-7bc5f796dc-gxcgj\" (UID: \"b2d4dbd6-6084-46c1-a832-3f09ed3ab603\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a264e-predictor-7bc5f796dc-gxcgj"
Sep 17 23:19:56.329190 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:19:56.329174    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df6589c46-cx4bd"
Sep 17 23:19:56.329603 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:19:56.329581    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b2d4dbd6-6084-46c1-a832-3f09ed3ab603-proxy-tls\") pod \"isvc-raw-sklearn-batcher-a264e-predictor-7bc5f796dc-gxcgj\" (UID: \"b2d4dbd6-6084-46c1-a832-3f09ed3ab603\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a264e-predictor-7bc5f796dc-gxcgj"
Sep 17 23:19:56.331595 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:19:56.331576    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-predictor-serving-cert\""
Sep 17 23:19:56.331685 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:19:56.331599    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-kube-rbac-proxy-sar-config\""
Sep 17 23:19:56.333159 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:19:56.333136    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:19:56.336944 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:19:56.336915    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-khjgc\" (UniqueName: \"kubernetes.io/projected/b2d4dbd6-6084-46c1-a832-3f09ed3ab603-kube-api-access-khjgc\") pod \"isvc-raw-sklearn-batcher-a264e-predictor-7bc5f796dc-gxcgj\" (UID: \"b2d4dbd6-6084-46c1-a832-3f09ed3ab603\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a264e-predictor-7bc5f796dc-gxcgj"
Sep 17 23:19:56.341325 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:19:56.341305    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df6589c46-cx4bd"]
Sep 17 23:19:56.427610 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:19:56.427584    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ftbv2\" (UniqueName: \"kubernetes.io/projected/afdeceb7-80fb-4e2e-8493-3c113ee34e82-kube-api-access-ftbv2\") pod \"isvc-canary-rollback-predictor-7df6589c46-cx4bd\" (UID: \"afdeceb7-80fb-4e2e-8493-3c113ee34e82\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df6589c46-cx4bd"
Sep 17 23:19:56.427702 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:19:56.427614    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/afdeceb7-80fb-4e2e-8493-3c113ee34e82-proxy-tls\") pod \"isvc-canary-rollback-predictor-7df6589c46-cx4bd\" (UID: \"afdeceb7-80fb-4e2e-8493-3c113ee34e82\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df6589c46-cx4bd"
Sep 17 23:19:56.427702 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:19:56.427646    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/afdeceb7-80fb-4e2e-8493-3c113ee34e82-cabundle-cert\") pod \"isvc-canary-rollback-predictor-7df6589c46-cx4bd\" (UID: \"afdeceb7-80fb-4e2e-8493-3c113ee34e82\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df6589c46-cx4bd"
Sep 17 23:19:56.427834 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:19:56.427716    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/afdeceb7-80fb-4e2e-8493-3c113ee34e82-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-7df6589c46-cx4bd\" (UID: \"afdeceb7-80fb-4e2e-8493-3c113ee34e82\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df6589c46-cx4bd"
Sep 17 23:19:56.427834 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:19:56.427761    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/afdeceb7-80fb-4e2e-8493-3c113ee34e82-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-7df6589c46-cx4bd\" (UID: \"afdeceb7-80fb-4e2e-8493-3c113ee34e82\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df6589c46-cx4bd"
Sep 17 23:19:56.433120 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:19:56.433100    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a264e-predictor-7bc5f796dc-gxcgj"
Sep 17 23:19:56.507336 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:19:56.507308    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:19:56.528320 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:19:56.528288    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/afdeceb7-80fb-4e2e-8493-3c113ee34e82-cabundle-cert\") pod \"isvc-canary-rollback-predictor-7df6589c46-cx4bd\" (UID: \"afdeceb7-80fb-4e2e-8493-3c113ee34e82\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df6589c46-cx4bd"
Sep 17 23:19:56.528435 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:19:56.528351    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/afdeceb7-80fb-4e2e-8493-3c113ee34e82-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-7df6589c46-cx4bd\" (UID: \"afdeceb7-80fb-4e2e-8493-3c113ee34e82\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df6589c46-cx4bd"
Sep 17 23:19:56.528435 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:19:56.528421    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/afdeceb7-80fb-4e2e-8493-3c113ee34e82-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-7df6589c46-cx4bd\" (UID: \"afdeceb7-80fb-4e2e-8493-3c113ee34e82\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df6589c46-cx4bd"
Sep 17 23:19:56.528542 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:19:56.528468    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ftbv2\" (UniqueName: \"kubernetes.io/projected/afdeceb7-80fb-4e2e-8493-3c113ee34e82-kube-api-access-ftbv2\") pod \"isvc-canary-rollback-predictor-7df6589c46-cx4bd\" (UID: \"afdeceb7-80fb-4e2e-8493-3c113ee34e82\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df6589c46-cx4bd"
Sep 17 23:19:56.528542 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:19:56.528504    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/afdeceb7-80fb-4e2e-8493-3c113ee34e82-proxy-tls\") pod \"isvc-canary-rollback-predictor-7df6589c46-cx4bd\" (UID: \"afdeceb7-80fb-4e2e-8493-3c113ee34e82\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df6589c46-cx4bd"
Sep 17 23:19:56.528788 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:19:56.528767    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/afdeceb7-80fb-4e2e-8493-3c113ee34e82-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-7df6589c46-cx4bd\" (UID: \"afdeceb7-80fb-4e2e-8493-3c113ee34e82\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df6589c46-cx4bd"
Sep 17 23:19:56.529045 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:19:56.529028    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/afdeceb7-80fb-4e2e-8493-3c113ee34e82-cabundle-cert\") pod \"isvc-canary-rollback-predictor-7df6589c46-cx4bd\" (UID: \"afdeceb7-80fb-4e2e-8493-3c113ee34e82\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df6589c46-cx4bd"
Sep 17 23:19:56.529108 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:19:56.529048    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/afdeceb7-80fb-4e2e-8493-3c113ee34e82-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-7df6589c46-cx4bd\" (UID: \"afdeceb7-80fb-4e2e-8493-3c113ee34e82\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df6589c46-cx4bd"
Sep 17 23:19:56.531038 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:19:56.530997    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/afdeceb7-80fb-4e2e-8493-3c113ee34e82-proxy-tls\") pod \"isvc-canary-rollback-predictor-7df6589c46-cx4bd\" (UID: \"afdeceb7-80fb-4e2e-8493-3c113ee34e82\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df6589c46-cx4bd"
Sep 17 23:19:56.536645 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:19:56.536618    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ftbv2\" (UniqueName: \"kubernetes.io/projected/afdeceb7-80fb-4e2e-8493-3c113ee34e82-kube-api-access-ftbv2\") pod \"isvc-canary-rollback-predictor-7df6589c46-cx4bd\" (UID: \"afdeceb7-80fb-4e2e-8493-3c113ee34e82\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df6589c46-cx4bd"
Sep 17 23:19:56.632313 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:19:56.632251    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:19:56.652152 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:19:56.652127    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df6589c46-cx4bd"
Sep 17 23:19:56.709305 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:19:56.709246    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a264e-predictor-7bc5f796dc-gxcgj"]
Sep 17 23:19:56.710027 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:19:56.709977    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2d4dbd6_6084_46c1_a832_3f09ed3ab603.slice/crio-3ab06b6d7ebc97b95b7ea664e741c9acd4c604558865f2fd16aea56d84eb89a8 WatchSource:0}: Error finding container 3ab06b6d7ebc97b95b7ea664e741c9acd4c604558865f2fd16aea56d84eb89a8: Status 404 returned error can't find the container with id 3ab06b6d7ebc97b95b7ea664e741c9acd4c604558865f2fd16aea56d84eb89a8
Sep 17 23:19:56.926358 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:19:56.926331    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df6589c46-cx4bd"]
Sep 17 23:19:56.927604 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:19:56.927580    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podafdeceb7_80fb_4e2e_8493_3c113ee34e82.slice/crio-4b0fc6ff22b200876bcfdd8325495b72bda217aae7ef20c85fbaa7207ae074db WatchSource:0}: Error finding container 4b0fc6ff22b200876bcfdd8325495b72bda217aae7ef20c85fbaa7207ae074db: Status 404 returned error can't find the container with id 4b0fc6ff22b200876bcfdd8325495b72bda217aae7ef20c85fbaa7207ae074db
Sep 17 23:19:57.184974 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:19:57.184876    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a264e-predictor-7bc5f796dc-gxcgj" event={"ID":"b2d4dbd6-6084-46c1-a832-3f09ed3ab603","Type":"ContainerStarted","Data":"d5129d8c5a2aee32b2b1a6997992d3d35664c2a520df3f719e6788af035ddf6f"}
Sep 17 23:19:57.185419 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:19:57.184976    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a264e-predictor-7bc5f796dc-gxcgj" event={"ID":"b2d4dbd6-6084-46c1-a832-3f09ed3ab603","Type":"ContainerStarted","Data":"3ab06b6d7ebc97b95b7ea664e741c9acd4c604558865f2fd16aea56d84eb89a8"}
Sep 17 23:19:57.186810 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:19:57.186774    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df6589c46-cx4bd" event={"ID":"afdeceb7-80fb-4e2e-8493-3c113ee34e82","Type":"ContainerStarted","Data":"2d76b1de4ceb8e5c8015e857ed1b45120ddff47bc888e2b262aff2ae8af36fcf"}
Sep 17 23:19:57.186920 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:19:57.186834    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df6589c46-cx4bd" event={"ID":"afdeceb7-80fb-4e2e-8493-3c113ee34e82","Type":"ContainerStarted","Data":"4b0fc6ff22b200876bcfdd8325495b72bda217aae7ef20c85fbaa7207ae074db"}
Sep 17 23:20:01.204963 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:20:01.204927    2584 generic.go:356] "Generic (PLEG): container finished" podID="b2d4dbd6-6084-46c1-a832-3f09ed3ab603" containerID="d5129d8c5a2aee32b2b1a6997992d3d35664c2a520df3f719e6788af035ddf6f" exitCode=0
Sep 17 23:20:01.205392 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:20:01.205019    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a264e-predictor-7bc5f796dc-gxcgj" event={"ID":"b2d4dbd6-6084-46c1-a832-3f09ed3ab603","Type":"ContainerDied","Data":"d5129d8c5a2aee32b2b1a6997992d3d35664c2a520df3f719e6788af035ddf6f"}
Sep 17 23:20:01.206684 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:20:01.206663    2584 generic.go:356] "Generic (PLEG): container finished" podID="afdeceb7-80fb-4e2e-8493-3c113ee34e82" containerID="2d76b1de4ceb8e5c8015e857ed1b45120ddff47bc888e2b262aff2ae8af36fcf" exitCode=0
Sep 17 23:20:01.206785 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:20:01.206692    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df6589c46-cx4bd" event={"ID":"afdeceb7-80fb-4e2e-8493-3c113ee34e82","Type":"ContainerDied","Data":"2d76b1de4ceb8e5c8015e857ed1b45120ddff47bc888e2b262aff2ae8af36fcf"}
Sep 17 23:20:18.318045 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:20:18.317986    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a264e-predictor-7bc5f796dc-gxcgj" event={"ID":"b2d4dbd6-6084-46c1-a832-3f09ed3ab603","Type":"ContainerStarted","Data":"525ada3b3b7ee44db5e2f7d08eaec7c1e9c4b7f5e07f0d5751da9ccd02a2183e"}
Sep 17 23:20:18.321847 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:20:18.321780    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df6589c46-cx4bd" event={"ID":"afdeceb7-80fb-4e2e-8493-3c113ee34e82","Type":"ContainerStarted","Data":"25c7ae2dc5db70b21eb33122c6f25ab729ffeb867b164a5ff69f0ef292ff2d68"}
Sep 17 23:20:21.339564 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:20:21.339522    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df6589c46-cx4bd" event={"ID":"afdeceb7-80fb-4e2e-8493-3c113ee34e82","Type":"ContainerStarted","Data":"0c442e0005c2af8e281075bd6cd901455ed70729745624b34434e03bca8274ea"}
Sep 17 23:20:21.340075 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:20:21.339717    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df6589c46-cx4bd"
Sep 17 23:20:21.340075 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:20:21.339864    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df6589c46-cx4bd"
Sep 17 23:20:21.341332 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:20:21.341302    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df6589c46-cx4bd" podUID="afdeceb7-80fb-4e2e-8493-3c113ee34e82" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.21:8080: connect: connection refused"
Sep 17 23:20:21.342281 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:20:21.342256    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a264e-predictor-7bc5f796dc-gxcgj" event={"ID":"b2d4dbd6-6084-46c1-a832-3f09ed3ab603","Type":"ContainerStarted","Data":"ce6547daf7517f2b63e0a174d6a2e0727c235476ed392a4cad651431f687cb1c"}
Sep 17 23:20:21.361134 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:20:21.361093    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df6589c46-cx4bd" podStartSLOduration=2.229378192 podStartE2EDuration="25.361078515s" totalImagesPullingTime="19.207734466s" totalInitContainerRuntime="3.923965857s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:19:56 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:20:21.358260426 +0000 UTC m=+604.723920343" watchObservedRunningTime="2026-09-17 23:20:21.361078515 +0000 UTC m=+604.726738446"
Sep 17 23:20:22.347741 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:20:22.347664    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df6589c46-cx4bd" podUID="afdeceb7-80fb-4e2e-8493-3c113ee34e82" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.21:8080: connect: connection refused"
Sep 17 23:20:23.354070 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:20:23.353968    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a264e-predictor-7bc5f796dc-gxcgj" event={"ID":"b2d4dbd6-6084-46c1-a832-3f09ed3ab603","Type":"ContainerStarted","Data":"a07ee5c944dee04a566cbcdb35488095c6a165382a79820632bc5b5134bdca77"}
Sep 17 23:20:23.354513 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:20:23.354245    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a264e-predictor-7bc5f796dc-gxcgj"
Sep 17 23:20:23.354513 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:20:23.354383    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a264e-predictor-7bc5f796dc-gxcgj"
Sep 17 23:20:23.356228 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:20:23.356202    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a264e-predictor-7bc5f796dc-gxcgj" podUID="b2d4dbd6-6084-46c1-a832-3f09ed3ab603" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 23:20:23.375427 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:20:23.375387    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a264e-predictor-7bc5f796dc-gxcgj" podStartSLOduration=1.839999901 podStartE2EDuration="27.375375069s" totalImagesPullingTime="21.590613527s" totalInitContainerRuntime="3.944761641s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:19:56 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:20:23.375148931 +0000 UTC m=+606.740808848" watchObservedRunningTime="2026-09-17 23:20:23.375375069 +0000 UTC m=+606.741034985"
Sep 17 23:20:24.358022 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:20:24.357975    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a264e-predictor-7bc5f796dc-gxcgj"
Sep 17 23:20:24.358530 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:20:24.358428    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a264e-predictor-7bc5f796dc-gxcgj" podUID="b2d4dbd6-6084-46c1-a832-3f09ed3ab603" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 23:20:24.358530 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:20:24.358432    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a264e-predictor-7bc5f796dc-gxcgj" podUID="b2d4dbd6-6084-46c1-a832-3f09ed3ab603" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.20:8080: connect: connection refused"
Sep 17 23:20:25.361913 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:20:25.361866    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a264e-predictor-7bc5f796dc-gxcgj" podUID="b2d4dbd6-6084-46c1-a832-3f09ed3ab603" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.20:8080: connect: connection refused"
Sep 17 23:20:25.362356 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:20:25.362210    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a264e-predictor-7bc5f796dc-gxcgj" podUID="b2d4dbd6-6084-46c1-a832-3f09ed3ab603" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 23:20:25.365692 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:20:25.365673    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a264e-predictor-7bc5f796dc-gxcgj"
Sep 17 23:20:26.365539 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:20:26.365500    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a264e-predictor-7bc5f796dc-gxcgj" podUID="b2d4dbd6-6084-46c1-a832-3f09ed3ab603" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.20:8080: connect: connection refused"
Sep 17 23:20:26.365986 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:20:26.365840    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a264e-predictor-7bc5f796dc-gxcgj" podUID="b2d4dbd6-6084-46c1-a832-3f09ed3ab603" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 23:20:27.352103 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:20:27.352074    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df6589c46-cx4bd"
Sep 17 23:20:27.352656 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:20:27.352629    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df6589c46-cx4bd" podUID="afdeceb7-80fb-4e2e-8493-3c113ee34e82" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.21:8080: connect: connection refused"
Sep 17 23:20:36.366184 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:20:36.366092    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a264e-predictor-7bc5f796dc-gxcgj" podUID="b2d4dbd6-6084-46c1-a832-3f09ed3ab603" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.20:8080: connect: connection refused"
Sep 17 23:20:36.366584 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:20:36.366459    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a264e-predictor-7bc5f796dc-gxcgj" podUID="b2d4dbd6-6084-46c1-a832-3f09ed3ab603" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 23:20:37.352659 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:20:37.352620    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df6589c46-cx4bd" podUID="afdeceb7-80fb-4e2e-8493-3c113ee34e82" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.21:8080: connect: connection refused"
Sep 17 23:20:46.365576 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:20:46.365531    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a264e-predictor-7bc5f796dc-gxcgj" podUID="b2d4dbd6-6084-46c1-a832-3f09ed3ab603" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.20:8080: connect: connection refused"
Sep 17 23:20:46.366070 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:20:46.366045    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a264e-predictor-7bc5f796dc-gxcgj" podUID="b2d4dbd6-6084-46c1-a832-3f09ed3ab603" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 23:20:47.353264 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:20:47.353223    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df6589c46-cx4bd" podUID="afdeceb7-80fb-4e2e-8493-3c113ee34e82" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.21:8080: connect: connection refused"
Sep 17 23:20:56.366394 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:20:56.366344    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a264e-predictor-7bc5f796dc-gxcgj" podUID="b2d4dbd6-6084-46c1-a832-3f09ed3ab603" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.20:8080: connect: connection refused"
Sep 17 23:20:56.366867 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:20:56.366768    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a264e-predictor-7bc5f796dc-gxcgj" podUID="b2d4dbd6-6084-46c1-a832-3f09ed3ab603" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 23:20:57.352816 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:20:57.352778    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df6589c46-cx4bd" podUID="afdeceb7-80fb-4e2e-8493-3c113ee34e82" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.21:8080: connect: connection refused"
Sep 17 23:21:06.366186 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:21:06.366136    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a264e-predictor-7bc5f796dc-gxcgj" podUID="b2d4dbd6-6084-46c1-a832-3f09ed3ab603" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.20:8080: connect: connection refused"
Sep 17 23:21:06.366632 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:21:06.366576    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a264e-predictor-7bc5f796dc-gxcgj" podUID="b2d4dbd6-6084-46c1-a832-3f09ed3ab603" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 23:21:07.353120 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:21:07.353075    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df6589c46-cx4bd" podUID="afdeceb7-80fb-4e2e-8493-3c113ee34e82" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.21:8080: connect: connection refused"
Sep 17 23:21:16.366394 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:21:16.366345    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a264e-predictor-7bc5f796dc-gxcgj" podUID="b2d4dbd6-6084-46c1-a832-3f09ed3ab603" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.20:8080: connect: connection refused"
Sep 17 23:21:16.366810 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:21:16.366787    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a264e-predictor-7bc5f796dc-gxcgj" podUID="b2d4dbd6-6084-46c1-a832-3f09ed3ab603" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 23:21:17.352736 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:21:17.352696    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df6589c46-cx4bd" podUID="afdeceb7-80fb-4e2e-8493-3c113ee34e82" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.21:8080: connect: connection refused"
Sep 17 23:21:26.366348 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:21:26.366301    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a264e-predictor-7bc5f796dc-gxcgj" podUID="b2d4dbd6-6084-46c1-a832-3f09ed3ab603" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.20:8080: connect: connection refused"
Sep 17 23:21:26.366780 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:21:26.366695    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a264e-predictor-7bc5f796dc-gxcgj" podUID="b2d4dbd6-6084-46c1-a832-3f09ed3ab603" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 23:21:27.352556 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:21:27.352507    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df6589c46-cx4bd" podUID="afdeceb7-80fb-4e2e-8493-3c113ee34e82" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.21:8080: connect: connection refused"
Sep 17 23:21:35.759121 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:21:35.759082    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a264e-predictor-7bc5f796dc-gxcgj"
Sep 17 23:21:35.759574 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:21:35.759178    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a264e-predictor-7bc5f796dc-gxcgj"
Sep 17 23:21:37.353180 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:21:37.353150    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df6589c46-cx4bd"
Sep 17 23:21:46.092897 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:21:46.092811    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:21:51.092198 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:21:51.092164    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a264e-predictor-7bc5f796dc-gxcgj"]
Sep 17 23:21:51.092602 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:21:51.092412    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a264e-predictor-7bc5f796dc-gxcgj" podUID="b2d4dbd6-6084-46c1-a832-3f09ed3ab603" containerName="kserve-container" containerID="cri-o://525ada3b3b7ee44db5e2f7d08eaec7c1e9c4b7f5e07f0d5751da9ccd02a2183e" gracePeriod=30
Sep 17 23:21:51.092602 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:21:51.092436    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a264e-predictor-7bc5f796dc-gxcgj" podUID="b2d4dbd6-6084-46c1-a832-3f09ed3ab603" containerName="agent" containerID="cri-o://a07ee5c944dee04a566cbcdb35488095c6a165382a79820632bc5b5134bdca77" gracePeriod=30
Sep 17 23:21:51.092602 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:21:51.092465    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a264e-predictor-7bc5f796dc-gxcgj" podUID="b2d4dbd6-6084-46c1-a832-3f09ed3ab603" containerName="kube-rbac-proxy" containerID="cri-o://ce6547daf7517f2b63e0a174d6a2e0727c235476ed392a4cad651431f687cb1c" gracePeriod=30
Sep 17 23:21:51.104786 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:21:51.104762    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:21:51.228800 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:21:51.228769    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:21:51.425460 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:21:51.425365    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:21:51.511593 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:21:51.511563    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:21:51.739814 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:21:51.739736    2584 generic.go:356] "Generic (PLEG): container finished" podID="b2d4dbd6-6084-46c1-a832-3f09ed3ab603" containerID="ce6547daf7517f2b63e0a174d6a2e0727c235476ed392a4cad651431f687cb1c" exitCode=2
Sep 17 23:21:51.739814 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:21:51.739776    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a264e-predictor-7bc5f796dc-gxcgj" event={"ID":"b2d4dbd6-6084-46c1-a832-3f09ed3ab603","Type":"ContainerDied","Data":"ce6547daf7517f2b63e0a174d6a2e0727c235476ed392a4cad651431f687cb1c"}
Sep 17 23:21:51.863660 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:21:51.863626    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:21:55.362082 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:21:55.362037    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a264e-predictor-7bc5f796dc-gxcgj" podUID="b2d4dbd6-6084-46c1-a832-3f09ed3ab603" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.20:8643/healthz\": dial tcp 10.132.0.20:8643: connect: connection refused"
Sep 17 23:21:55.758467 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:21:55.758419    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a264e-predictor-7bc5f796dc-gxcgj" podUID="b2d4dbd6-6084-46c1-a832-3f09ed3ab603" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.20:8080: connect: connection refused"
Sep 17 23:21:55.758735 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:21:55.758705    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a264e-predictor-7bc5f796dc-gxcgj" podUID="b2d4dbd6-6084-46c1-a832-3f09ed3ab603" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 23:21:56.764820 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:21:56.764783    2584 generic.go:356] "Generic (PLEG): container finished" podID="b2d4dbd6-6084-46c1-a832-3f09ed3ab603" containerID="525ada3b3b7ee44db5e2f7d08eaec7c1e9c4b7f5e07f0d5751da9ccd02a2183e" exitCode=0
Sep 17 23:21:56.765234 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:21:56.764852    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a264e-predictor-7bc5f796dc-gxcgj" event={"ID":"b2d4dbd6-6084-46c1-a832-3f09ed3ab603","Type":"ContainerDied","Data":"525ada3b3b7ee44db5e2f7d08eaec7c1e9c4b7f5e07f0d5751da9ccd02a2183e"}
Sep 17 23:22:00.362570 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:22:00.362491    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a264e-predictor-7bc5f796dc-gxcgj" podUID="b2d4dbd6-6084-46c1-a832-3f09ed3ab603" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.20:8643/healthz\": dial tcp 10.132.0.20:8643: connect: connection refused"
Sep 17 23:22:02.224523 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:22:02.224485    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:22:05.362509 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:22:05.362461    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a264e-predictor-7bc5f796dc-gxcgj" podUID="b2d4dbd6-6084-46c1-a832-3f09ed3ab603" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.20:8643/healthz\": dial tcp 10.132.0.20:8643: connect: connection refused"
Sep 17 23:22:05.362953 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:22:05.362610    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a264e-predictor-7bc5f796dc-gxcgj"
Sep 17 23:22:05.758583 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:22:05.758537    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a264e-predictor-7bc5f796dc-gxcgj" podUID="b2d4dbd6-6084-46c1-a832-3f09ed3ab603" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.20:8080: connect: connection refused"
Sep 17 23:22:05.758795 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:22:05.758769    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a264e-predictor-7bc5f796dc-gxcgj" podUID="b2d4dbd6-6084-46c1-a832-3f09ed3ab603" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 23:22:10.361952 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:22:10.361916    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a264e-predictor-7bc5f796dc-gxcgj" podUID="b2d4dbd6-6084-46c1-a832-3f09ed3ab603" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.20:8643/healthz\": dial tcp 10.132.0.20:8643: connect: connection refused"
Sep 17 23:22:15.362435 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:22:15.362390    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a264e-predictor-7bc5f796dc-gxcgj" podUID="b2d4dbd6-6084-46c1-a832-3f09ed3ab603" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.20:8643/healthz\": dial tcp 10.132.0.20:8643: connect: connection refused"
Sep 17 23:22:15.759153 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:22:15.759109    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a264e-predictor-7bc5f796dc-gxcgj" podUID="b2d4dbd6-6084-46c1-a832-3f09ed3ab603" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.20:8080: connect: connection refused"
Sep 17 23:22:15.759340 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:22:15.759239    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a264e-predictor-7bc5f796dc-gxcgj"
Sep 17 23:22:15.759479 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:22:15.759460    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a264e-predictor-7bc5f796dc-gxcgj" podUID="b2d4dbd6-6084-46c1-a832-3f09ed3ab603" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 23:22:15.759560 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:22:15.759549    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a264e-predictor-7bc5f796dc-gxcgj"
Sep 17 23:22:20.362577 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:22:20.362522    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a264e-predictor-7bc5f796dc-gxcgj" podUID="b2d4dbd6-6084-46c1-a832-3f09ed3ab603" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.20:8643/healthz\": dial tcp 10.132.0.20:8643: connect: connection refused"
Sep 17 23:22:21.227477 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:22:21.227452    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a264e-predictor-7bc5f796dc-gxcgj"
Sep 17 23:22:21.370989 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:22:21.370915    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b2d4dbd6-6084-46c1-a832-3f09ed3ab603-proxy-tls\") pod \"b2d4dbd6-6084-46c1-a832-3f09ed3ab603\" (UID: \"b2d4dbd6-6084-46c1-a832-3f09ed3ab603\") "
Sep 17 23:22:21.370989 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:22:21.370951    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b2d4dbd6-6084-46c1-a832-3f09ed3ab603-cabundle-cert\") pod \"b2d4dbd6-6084-46c1-a832-3f09ed3ab603\" (UID: \"b2d4dbd6-6084-46c1-a832-3f09ed3ab603\") "
Sep 17 23:22:21.370989 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:22:21.370978    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-khjgc\" (UniqueName: \"kubernetes.io/projected/b2d4dbd6-6084-46c1-a832-3f09ed3ab603-kube-api-access-khjgc\") pod \"b2d4dbd6-6084-46c1-a832-3f09ed3ab603\" (UID: \"b2d4dbd6-6084-46c1-a832-3f09ed3ab603\") "
Sep 17 23:22:21.371494 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:22:21.371049    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b2d4dbd6-6084-46c1-a832-3f09ed3ab603-kube-rbac-proxy-sar-config\") pod \"b2d4dbd6-6084-46c1-a832-3f09ed3ab603\" (UID: \"b2d4dbd6-6084-46c1-a832-3f09ed3ab603\") "
Sep 17 23:22:21.371494 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:22:21.371067    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b2d4dbd6-6084-46c1-a832-3f09ed3ab603-kserve-provision-location\") pod \"b2d4dbd6-6084-46c1-a832-3f09ed3ab603\" (UID: \"b2d4dbd6-6084-46c1-a832-3f09ed3ab603\") "
Sep 17 23:22:21.371494 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:22:21.371305    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2d4dbd6-6084-46c1-a832-3f09ed3ab603-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "b2d4dbd6-6084-46c1-a832-3f09ed3ab603" (UID: "b2d4dbd6-6084-46c1-a832-3f09ed3ab603"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:22:21.371494 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:22:21.371367    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2d4dbd6-6084-46c1-a832-3f09ed3ab603-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "b2d4dbd6-6084-46c1-a832-3f09ed3ab603" (UID: "b2d4dbd6-6084-46c1-a832-3f09ed3ab603"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:22:21.371494 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:22:21.371373    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2d4dbd6-6084-46c1-a832-3f09ed3ab603-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "b2d4dbd6-6084-46c1-a832-3f09ed3ab603" (UID: "b2d4dbd6-6084-46c1-a832-3f09ed3ab603"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:22:21.373353 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:22:21.373329    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2d4dbd6-6084-46c1-a832-3f09ed3ab603-kube-api-access-khjgc" (OuterVolumeSpecName: "kube-api-access-khjgc") pod "b2d4dbd6-6084-46c1-a832-3f09ed3ab603" (UID: "b2d4dbd6-6084-46c1-a832-3f09ed3ab603"). InnerVolumeSpecName "kube-api-access-khjgc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:22:21.373353 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:22:21.373330    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b2d4dbd6-6084-46c1-a832-3f09ed3ab603-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "b2d4dbd6-6084-46c1-a832-3f09ed3ab603" (UID: "b2d4dbd6-6084-46c1-a832-3f09ed3ab603"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:22:21.471865 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:22:21.471837    2584 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b2d4dbd6-6084-46c1-a832-3f09ed3ab603-cabundle-cert\") on node \"ip-10-0-142-4.ec2.internal\" DevicePath \"\""
Sep 17 23:22:21.471865 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:22:21.471861    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-khjgc\" (UniqueName: \"kubernetes.io/projected/b2d4dbd6-6084-46c1-a832-3f09ed3ab603-kube-api-access-khjgc\") on node \"ip-10-0-142-4.ec2.internal\" DevicePath \"\""
Sep 17 23:22:21.472050 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:22:21.471876    2584 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b2d4dbd6-6084-46c1-a832-3f09ed3ab603-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-4.ec2.internal\" DevicePath \"\""
Sep 17 23:22:21.472050 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:22:21.471890    2584 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b2d4dbd6-6084-46c1-a832-3f09ed3ab603-kserve-provision-location\") on node \"ip-10-0-142-4.ec2.internal\" DevicePath \"\""
Sep 17 23:22:21.472050 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:22:21.471901    2584 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b2d4dbd6-6084-46c1-a832-3f09ed3ab603-proxy-tls\") on node \"ip-10-0-142-4.ec2.internal\" DevicePath \"\""
Sep 17 23:22:21.867690 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:22:21.867658    2584 generic.go:356] "Generic (PLEG): container finished" podID="b2d4dbd6-6084-46c1-a832-3f09ed3ab603" containerID="a07ee5c944dee04a566cbcdb35488095c6a165382a79820632bc5b5134bdca77" exitCode=0
Sep 17 23:22:21.867870 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:22:21.867724    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a264e-predictor-7bc5f796dc-gxcgj"
Sep 17 23:22:21.867870 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:22:21.867745    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a264e-predictor-7bc5f796dc-gxcgj" event={"ID":"b2d4dbd6-6084-46c1-a832-3f09ed3ab603","Type":"ContainerDied","Data":"a07ee5c944dee04a566cbcdb35488095c6a165382a79820632bc5b5134bdca77"}
Sep 17 23:22:21.867870 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:22:21.867813    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a264e-predictor-7bc5f796dc-gxcgj" event={"ID":"b2d4dbd6-6084-46c1-a832-3f09ed3ab603","Type":"ContainerDied","Data":"3ab06b6d7ebc97b95b7ea664e741c9acd4c604558865f2fd16aea56d84eb89a8"}
Sep 17 23:22:21.867870 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:22:21.867846    2584 scope.go:117] "RemoveContainer" containerID="a07ee5c944dee04a566cbcdb35488095c6a165382a79820632bc5b5134bdca77"
Sep 17 23:22:21.876584 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:22:21.876568    2584 scope.go:117] "RemoveContainer" containerID="ce6547daf7517f2b63e0a174d6a2e0727c235476ed392a4cad651431f687cb1c"
Sep 17 23:22:21.884303 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:22:21.884285    2584 scope.go:117] "RemoveContainer" containerID="525ada3b3b7ee44db5e2f7d08eaec7c1e9c4b7f5e07f0d5751da9ccd02a2183e"
Sep 17 23:22:21.890298 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:22:21.890277    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a264e-predictor-7bc5f796dc-gxcgj"]
Sep 17 23:22:21.892161 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:22:21.892134    2584 scope.go:117] "RemoveContainer" containerID="d5129d8c5a2aee32b2b1a6997992d3d35664c2a520df3f719e6788af035ddf6f"
Sep 17 23:22:21.897684 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:22:21.897662    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a264e-predictor-7bc5f796dc-gxcgj"]
Sep 17 23:22:21.899795 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:22:21.899781    2584 scope.go:117] "RemoveContainer" containerID="a07ee5c944dee04a566cbcdb35488095c6a165382a79820632bc5b5134bdca77"
Sep 17 23:22:21.900098 ip-10-0-142-4 kubenswrapper[2584]: E0917 23:22:21.900079    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a07ee5c944dee04a566cbcdb35488095c6a165382a79820632bc5b5134bdca77\": container with ID starting with a07ee5c944dee04a566cbcdb35488095c6a165382a79820632bc5b5134bdca77 not found: ID does not exist" containerID="a07ee5c944dee04a566cbcdb35488095c6a165382a79820632bc5b5134bdca77"
Sep 17 23:22:21.900156 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:22:21.900106    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a07ee5c944dee04a566cbcdb35488095c6a165382a79820632bc5b5134bdca77"} err="failed to get container status \"a07ee5c944dee04a566cbcdb35488095c6a165382a79820632bc5b5134bdca77\": rpc error: code = NotFound desc = could not find container \"a07ee5c944dee04a566cbcdb35488095c6a165382a79820632bc5b5134bdca77\": container with ID starting with a07ee5c944dee04a566cbcdb35488095c6a165382a79820632bc5b5134bdca77 not found: ID does not exist"
Sep 17 23:22:21.900156 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:22:21.900124    2584 scope.go:117] "RemoveContainer" containerID="ce6547daf7517f2b63e0a174d6a2e0727c235476ed392a4cad651431f687cb1c"
Sep 17 23:22:21.900365 ip-10-0-142-4 kubenswrapper[2584]: E0917 23:22:21.900347    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce6547daf7517f2b63e0a174d6a2e0727c235476ed392a4cad651431f687cb1c\": container with ID starting with ce6547daf7517f2b63e0a174d6a2e0727c235476ed392a4cad651431f687cb1c not found: ID does not exist" containerID="ce6547daf7517f2b63e0a174d6a2e0727c235476ed392a4cad651431f687cb1c"
Sep 17 23:22:21.900427 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:22:21.900374    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce6547daf7517f2b63e0a174d6a2e0727c235476ed392a4cad651431f687cb1c"} err="failed to get container status \"ce6547daf7517f2b63e0a174d6a2e0727c235476ed392a4cad651431f687cb1c\": rpc error: code = NotFound desc = could not find container \"ce6547daf7517f2b63e0a174d6a2e0727c235476ed392a4cad651431f687cb1c\": container with ID starting with ce6547daf7517f2b63e0a174d6a2e0727c235476ed392a4cad651431f687cb1c not found: ID does not exist"
Sep 17 23:22:21.900427 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:22:21.900396    2584 scope.go:117] "RemoveContainer" containerID="525ada3b3b7ee44db5e2f7d08eaec7c1e9c4b7f5e07f0d5751da9ccd02a2183e"
Sep 17 23:22:21.900637 ip-10-0-142-4 kubenswrapper[2584]: E0917 23:22:21.900621    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"525ada3b3b7ee44db5e2f7d08eaec7c1e9c4b7f5e07f0d5751da9ccd02a2183e\": container with ID starting with 525ada3b3b7ee44db5e2f7d08eaec7c1e9c4b7f5e07f0d5751da9ccd02a2183e not found: ID does not exist" containerID="525ada3b3b7ee44db5e2f7d08eaec7c1e9c4b7f5e07f0d5751da9ccd02a2183e"
Sep 17 23:22:21.900679 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:22:21.900641    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"525ada3b3b7ee44db5e2f7d08eaec7c1e9c4b7f5e07f0d5751da9ccd02a2183e"} err="failed to get container status \"525ada3b3b7ee44db5e2f7d08eaec7c1e9c4b7f5e07f0d5751da9ccd02a2183e\": rpc error: code = NotFound desc = could not find container \"525ada3b3b7ee44db5e2f7d08eaec7c1e9c4b7f5e07f0d5751da9ccd02a2183e\": container with ID starting with 525ada3b3b7ee44db5e2f7d08eaec7c1e9c4b7f5e07f0d5751da9ccd02a2183e not found: ID does not exist"
Sep 17 23:22:21.900679 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:22:21.900655    2584 scope.go:117] "RemoveContainer" containerID="d5129d8c5a2aee32b2b1a6997992d3d35664c2a520df3f719e6788af035ddf6f"
Sep 17 23:22:21.900885 ip-10-0-142-4 kubenswrapper[2584]: E0917 23:22:21.900868    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d5129d8c5a2aee32b2b1a6997992d3d35664c2a520df3f719e6788af035ddf6f\": container with ID starting with d5129d8c5a2aee32b2b1a6997992d3d35664c2a520df3f719e6788af035ddf6f not found: ID does not exist" containerID="d5129d8c5a2aee32b2b1a6997992d3d35664c2a520df3f719e6788af035ddf6f"
Sep 17 23:22:21.900937 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:22:21.900893    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5129d8c5a2aee32b2b1a6997992d3d35664c2a520df3f719e6788af035ddf6f"} err="failed to get container status \"d5129d8c5a2aee32b2b1a6997992d3d35664c2a520df3f719e6788af035ddf6f\": rpc error: code = NotFound desc = could not find container \"d5129d8c5a2aee32b2b1a6997992d3d35664c2a520df3f719e6788af035ddf6f\": container with ID starting with d5129d8c5a2aee32b2b1a6997992d3d35664c2a520df3f719e6788af035ddf6f not found: ID does not exist"
Sep 17 23:22:22.763622 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:22:22.763592    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b2d4dbd6-6084-46c1-a832-3f09ed3ab603" path="/var/lib/kubelet/pods/b2d4dbd6-6084-46c1-a832-3f09ed3ab603/volumes"
Sep 17 23:23:16.173274 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:23:16.173238    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df6589c46-cx4bd"]
Sep 17 23:23:16.173739 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:23:16.173510    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df6589c46-cx4bd" podUID="afdeceb7-80fb-4e2e-8493-3c113ee34e82" containerName="kserve-container" containerID="cri-o://25c7ae2dc5db70b21eb33122c6f25ab729ffeb867b164a5ff69f0ef292ff2d68" gracePeriod=30
Sep 17 23:23:16.173739 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:23:16.173539    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df6589c46-cx4bd" podUID="afdeceb7-80fb-4e2e-8493-3c113ee34e82" containerName="kube-rbac-proxy" containerID="cri-o://0c442e0005c2af8e281075bd6cd901455ed70729745624b34434e03bca8274ea" gracePeriod=30
Sep 17 23:23:16.197959 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:23:16.197927    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:23:17.090873 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:23:17.090839    2584 generic.go:356] "Generic (PLEG): container finished" podID="afdeceb7-80fb-4e2e-8493-3c113ee34e82" containerID="0c442e0005c2af8e281075bd6cd901455ed70729745624b34434e03bca8274ea" exitCode=2
Sep 17 23:23:17.091134 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:23:17.090912    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df6589c46-cx4bd" event={"ID":"afdeceb7-80fb-4e2e-8493-3c113ee34e82","Type":"ContainerDied","Data":"0c442e0005c2af8e281075bd6cd901455ed70729745624b34434e03bca8274ea"}
Sep 17 23:23:17.348682 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:23:17.348590    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df6589c46-cx4bd" podUID="afdeceb7-80fb-4e2e-8493-3c113ee34e82" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.21:8643/healthz\": dial tcp 10.132.0.21:8643: connect: connection refused"
Sep 17 23:23:17.352987 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:23:17.352963    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df6589c46-cx4bd" podUID="afdeceb7-80fb-4e2e-8493-3c113ee34e82" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.21:8080: connect: connection refused"
Sep 17 23:23:20.410245 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:23:20.410219    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df6589c46-cx4bd"
Sep 17 23:23:20.523564 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:23:20.523536    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/afdeceb7-80fb-4e2e-8493-3c113ee34e82-proxy-tls\") pod \"afdeceb7-80fb-4e2e-8493-3c113ee34e82\" (UID: \"afdeceb7-80fb-4e2e-8493-3c113ee34e82\") "
Sep 17 23:23:20.523705 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:23:20.523594    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/afdeceb7-80fb-4e2e-8493-3c113ee34e82-kube-rbac-proxy-sar-config\") pod \"afdeceb7-80fb-4e2e-8493-3c113ee34e82\" (UID: \"afdeceb7-80fb-4e2e-8493-3c113ee34e82\") "
Sep 17 23:23:20.523705 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:23:20.523659    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ftbv2\" (UniqueName: \"kubernetes.io/projected/afdeceb7-80fb-4e2e-8493-3c113ee34e82-kube-api-access-ftbv2\") pod \"afdeceb7-80fb-4e2e-8493-3c113ee34e82\" (UID: \"afdeceb7-80fb-4e2e-8493-3c113ee34e82\") "
Sep 17 23:23:20.523705 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:23:20.523684    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/afdeceb7-80fb-4e2e-8493-3c113ee34e82-cabundle-cert\") pod \"afdeceb7-80fb-4e2e-8493-3c113ee34e82\" (UID: \"afdeceb7-80fb-4e2e-8493-3c113ee34e82\") "
Sep 17 23:23:20.523821 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:23:20.523803    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/afdeceb7-80fb-4e2e-8493-3c113ee34e82-kserve-provision-location\") pod \"afdeceb7-80fb-4e2e-8493-3c113ee34e82\" (UID: \"afdeceb7-80fb-4e2e-8493-3c113ee34e82\") "
Sep 17 23:23:20.524095 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:23:20.524066    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/afdeceb7-80fb-4e2e-8493-3c113ee34e82-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "afdeceb7-80fb-4e2e-8493-3c113ee34e82" (UID: "afdeceb7-80fb-4e2e-8493-3c113ee34e82"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:23:20.524232 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:23:20.524096    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/afdeceb7-80fb-4e2e-8493-3c113ee34e82-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "afdeceb7-80fb-4e2e-8493-3c113ee34e82" (UID: "afdeceb7-80fb-4e2e-8493-3c113ee34e82"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:23:20.524232 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:23:20.524148    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afdeceb7-80fb-4e2e-8493-3c113ee34e82-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "afdeceb7-80fb-4e2e-8493-3c113ee34e82" (UID: "afdeceb7-80fb-4e2e-8493-3c113ee34e82"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:23:20.525824 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:23:20.525793    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/afdeceb7-80fb-4e2e-8493-3c113ee34e82-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "afdeceb7-80fb-4e2e-8493-3c113ee34e82" (UID: "afdeceb7-80fb-4e2e-8493-3c113ee34e82"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:23:20.525931 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:23:20.525911    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/afdeceb7-80fb-4e2e-8493-3c113ee34e82-kube-api-access-ftbv2" (OuterVolumeSpecName: "kube-api-access-ftbv2") pod "afdeceb7-80fb-4e2e-8493-3c113ee34e82" (UID: "afdeceb7-80fb-4e2e-8493-3c113ee34e82"). InnerVolumeSpecName "kube-api-access-ftbv2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:23:20.624802 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:23:20.624781    2584 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/afdeceb7-80fb-4e2e-8493-3c113ee34e82-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-4.ec2.internal\" DevicePath \"\""
Sep 17 23:23:20.624802 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:23:20.624801    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ftbv2\" (UniqueName: \"kubernetes.io/projected/afdeceb7-80fb-4e2e-8493-3c113ee34e82-kube-api-access-ftbv2\") on node \"ip-10-0-142-4.ec2.internal\" DevicePath \"\""
Sep 17 23:23:20.624976 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:23:20.624810    2584 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/afdeceb7-80fb-4e2e-8493-3c113ee34e82-cabundle-cert\") on node \"ip-10-0-142-4.ec2.internal\" DevicePath \"\""
Sep 17 23:23:20.624976 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:23:20.624819    2584 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/afdeceb7-80fb-4e2e-8493-3c113ee34e82-kserve-provision-location\") on node \"ip-10-0-142-4.ec2.internal\" DevicePath \"\""
Sep 17 23:23:20.624976 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:23:20.624827    2584 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/afdeceb7-80fb-4e2e-8493-3c113ee34e82-proxy-tls\") on node \"ip-10-0-142-4.ec2.internal\" DevicePath \"\""
Sep 17 23:23:21.109412 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:23:21.109324    2584 generic.go:356] "Generic (PLEG): container finished" podID="afdeceb7-80fb-4e2e-8493-3c113ee34e82" containerID="25c7ae2dc5db70b21eb33122c6f25ab729ffeb867b164a5ff69f0ef292ff2d68" exitCode=0
Sep 17 23:23:21.109412 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:23:21.109397    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df6589c46-cx4bd"
Sep 17 23:23:21.109583 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:23:21.109412    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df6589c46-cx4bd" event={"ID":"afdeceb7-80fb-4e2e-8493-3c113ee34e82","Type":"ContainerDied","Data":"25c7ae2dc5db70b21eb33122c6f25ab729ffeb867b164a5ff69f0ef292ff2d68"}
Sep 17 23:23:21.109583 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:23:21.109497    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df6589c46-cx4bd" event={"ID":"afdeceb7-80fb-4e2e-8493-3c113ee34e82","Type":"ContainerDied","Data":"4b0fc6ff22b200876bcfdd8325495b72bda217aae7ef20c85fbaa7207ae074db"}
Sep 17 23:23:21.109583 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:23:21.109524    2584 scope.go:117] "RemoveContainer" containerID="0c442e0005c2af8e281075bd6cd901455ed70729745624b34434e03bca8274ea"
Sep 17 23:23:21.117865 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:23:21.117846    2584 scope.go:117] "RemoveContainer" containerID="25c7ae2dc5db70b21eb33122c6f25ab729ffeb867b164a5ff69f0ef292ff2d68"
Sep 17 23:23:21.125037 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:23:21.125019    2584 scope.go:117] "RemoveContainer" containerID="2d76b1de4ceb8e5c8015e857ed1b45120ddff47bc888e2b262aff2ae8af36fcf"
Sep 17 23:23:21.128391 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:23:21.128368    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df6589c46-cx4bd"]
Sep 17 23:23:21.132778 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:23:21.132752    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df6589c46-cx4bd"]
Sep 17 23:23:21.133749 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:23:21.133733    2584 scope.go:117] "RemoveContainer" containerID="0c442e0005c2af8e281075bd6cd901455ed70729745624b34434e03bca8274ea"
Sep 17 23:23:21.134073 ip-10-0-142-4 kubenswrapper[2584]: E0917 23:23:21.134047    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c442e0005c2af8e281075bd6cd901455ed70729745624b34434e03bca8274ea\": container with ID starting with 0c442e0005c2af8e281075bd6cd901455ed70729745624b34434e03bca8274ea not found: ID does not exist" containerID="0c442e0005c2af8e281075bd6cd901455ed70729745624b34434e03bca8274ea"
Sep 17 23:23:21.134145 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:23:21.134085    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c442e0005c2af8e281075bd6cd901455ed70729745624b34434e03bca8274ea"} err="failed to get container status \"0c442e0005c2af8e281075bd6cd901455ed70729745624b34434e03bca8274ea\": rpc error: code = NotFound desc = could not find container \"0c442e0005c2af8e281075bd6cd901455ed70729745624b34434e03bca8274ea\": container with ID starting with 0c442e0005c2af8e281075bd6cd901455ed70729745624b34434e03bca8274ea not found: ID does not exist"
Sep 17 23:23:21.134145 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:23:21.134112    2584 scope.go:117] "RemoveContainer" containerID="25c7ae2dc5db70b21eb33122c6f25ab729ffeb867b164a5ff69f0ef292ff2d68"
Sep 17 23:23:21.134383 ip-10-0-142-4 kubenswrapper[2584]: E0917 23:23:21.134362    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"25c7ae2dc5db70b21eb33122c6f25ab729ffeb867b164a5ff69f0ef292ff2d68\": container with ID starting with 25c7ae2dc5db70b21eb33122c6f25ab729ffeb867b164a5ff69f0ef292ff2d68 not found: ID does not exist" containerID="25c7ae2dc5db70b21eb33122c6f25ab729ffeb867b164a5ff69f0ef292ff2d68"
Sep 17 23:23:21.134425 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:23:21.134390    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"25c7ae2dc5db70b21eb33122c6f25ab729ffeb867b164a5ff69f0ef292ff2d68"} err="failed to get container status \"25c7ae2dc5db70b21eb33122c6f25ab729ffeb867b164a5ff69f0ef292ff2d68\": rpc error: code = NotFound desc = could not find container \"25c7ae2dc5db70b21eb33122c6f25ab729ffeb867b164a5ff69f0ef292ff2d68\": container with ID starting with 25c7ae2dc5db70b21eb33122c6f25ab729ffeb867b164a5ff69f0ef292ff2d68 not found: ID does not exist"
Sep 17 23:23:21.134425 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:23:21.134412    2584 scope.go:117] "RemoveContainer" containerID="2d76b1de4ceb8e5c8015e857ed1b45120ddff47bc888e2b262aff2ae8af36fcf"
Sep 17 23:23:21.134659 ip-10-0-142-4 kubenswrapper[2584]: E0917 23:23:21.134643    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d76b1de4ceb8e5c8015e857ed1b45120ddff47bc888e2b262aff2ae8af36fcf\": container with ID starting with 2d76b1de4ceb8e5c8015e857ed1b45120ddff47bc888e2b262aff2ae8af36fcf not found: ID does not exist" containerID="2d76b1de4ceb8e5c8015e857ed1b45120ddff47bc888e2b262aff2ae8af36fcf"
Sep 17 23:23:21.134700 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:23:21.134666    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d76b1de4ceb8e5c8015e857ed1b45120ddff47bc888e2b262aff2ae8af36fcf"} err="failed to get container status \"2d76b1de4ceb8e5c8015e857ed1b45120ddff47bc888e2b262aff2ae8af36fcf\": rpc error: code = NotFound desc = could not find container \"2d76b1de4ceb8e5c8015e857ed1b45120ddff47bc888e2b262aff2ae8af36fcf\": container with ID starting with 2d76b1de4ceb8e5c8015e857ed1b45120ddff47bc888e2b262aff2ae8af36fcf not found: ID does not exist"
Sep 17 23:23:21.354869 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:23:21.354836    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:23:21.430394 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:23:21.430325    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:23:21.430790 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:23:21.430770    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-596476db66-ckjrl"]
Sep 17 23:23:21.431105 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:23:21.431092    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b2d4dbd6-6084-46c1-a832-3f09ed3ab603" containerName="kube-rbac-proxy"
Sep 17 23:23:21.431146 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:23:21.431107    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="b2d4dbd6-6084-46c1-a832-3f09ed3ab603" containerName="kube-rbac-proxy"
Sep 17 23:23:21.431146 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:23:21.431116    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b2d4dbd6-6084-46c1-a832-3f09ed3ab603" containerName="kserve-container"
Sep 17 23:23:21.431146 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:23:21.431123    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="b2d4dbd6-6084-46c1-a832-3f09ed3ab603" containerName="kserve-container"
Sep 17 23:23:21.431146 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:23:21.431129    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b2d4dbd6-6084-46c1-a832-3f09ed3ab603" containerName="agent"
Sep 17 23:23:21.431146 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:23:21.431134    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="b2d4dbd6-6084-46c1-a832-3f09ed3ab603" containerName="agent"
Sep 17 23:23:21.431146 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:23:21.431142    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="afdeceb7-80fb-4e2e-8493-3c113ee34e82" containerName="kube-rbac-proxy"
Sep 17 23:23:21.431146 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:23:21.431147    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="afdeceb7-80fb-4e2e-8493-3c113ee34e82" containerName="kube-rbac-proxy"
Sep 17 23:23:21.431349 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:23:21.431155    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b2d4dbd6-6084-46c1-a832-3f09ed3ab603" containerName="storage-initializer"
Sep 17 23:23:21.431349 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:23:21.431160    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="b2d4dbd6-6084-46c1-a832-3f09ed3ab603" containerName="storage-initializer"
Sep 17 23:23:21.431349 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:23:21.431177    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="afdeceb7-80fb-4e2e-8493-3c113ee34e82" containerName="kserve-container"
Sep 17 23:23:21.431349 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:23:21.431182    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="afdeceb7-80fb-4e2e-8493-3c113ee34e82" containerName="kserve-container"
Sep 17 23:23:21.431349 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:23:21.431188    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="afdeceb7-80fb-4e2e-8493-3c113ee34e82" containerName="storage-initializer"
Sep 17 23:23:21.431349 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:23:21.431193    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="afdeceb7-80fb-4e2e-8493-3c113ee34e82" containerName="storage-initializer"
Sep 17 23:23:21.431349 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:23:21.431239    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="afdeceb7-80fb-4e2e-8493-3c113ee34e82" containerName="kserve-container"
Sep 17 23:23:21.431349 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:23:21.431249    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="b2d4dbd6-6084-46c1-a832-3f09ed3ab603" containerName="kserve-container"
Sep 17 23:23:21.431349 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:23:21.431255    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="b2d4dbd6-6084-46c1-a832-3f09ed3ab603" containerName="kube-rbac-proxy"
Sep 17 23:23:21.431349 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:23:21.431261    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="afdeceb7-80fb-4e2e-8493-3c113ee34e82" containerName="kube-rbac-proxy"
Sep 17 23:23:21.431349 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:23:21.431266    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="b2d4dbd6-6084-46c1-a832-3f09ed3ab603" containerName="agent"
Sep 17 23:23:21.435850 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:23:21.435835    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-596476db66-ckjrl"
Sep 17 23:23:21.438330 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:23:21.438236    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 23:23:21.438330 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:23:21.438274    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 23:23:21.438330 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:23:21.438294    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-f2cbg\""
Sep 17 23:23:21.438330 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:23:21.438312    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 23:23:21.438649 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:23:21.438296    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-predictor-serving-cert\""
Sep 17 23:23:21.438649 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:23:21.438314    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 23:23:21.438649 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:23:21.438294    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-kube-rbac-proxy-sar-config\""
Sep 17 23:23:21.445506 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:23:21.445487    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-596476db66-ckjrl"]
Sep 17 23:23:21.506628 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:23:21.506600    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:23:21.531916 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:23:21.531889    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/afb89dd1-3bed-4626-981b-fdc7f7d1f3de-cabundle-cert\") pod \"isvc-canary-stop-predictor-596476db66-ckjrl\" (UID: \"afb89dd1-3bed-4626-981b-fdc7f7d1f3de\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-596476db66-ckjrl"
Sep 17 23:23:21.532060 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:23:21.531947    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/afb89dd1-3bed-4626-981b-fdc7f7d1f3de-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-596476db66-ckjrl\" (UID: \"afb89dd1-3bed-4626-981b-fdc7f7d1f3de\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-596476db66-ckjrl"
Sep 17 23:23:21.532060 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:23:21.532028    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r7j2m\" (UniqueName: \"kubernetes.io/projected/afb89dd1-3bed-4626-981b-fdc7f7d1f3de-kube-api-access-r7j2m\") pod \"isvc-canary-stop-predictor-596476db66-ckjrl\" (UID: \"afb89dd1-3bed-4626-981b-fdc7f7d1f3de\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-596476db66-ckjrl"
Sep 17 23:23:21.532157 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:23:21.532076    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/afb89dd1-3bed-4626-981b-fdc7f7d1f3de-proxy-tls\") pod \"isvc-canary-stop-predictor-596476db66-ckjrl\" (UID: \"afb89dd1-3bed-4626-981b-fdc7f7d1f3de\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-596476db66-ckjrl"
Sep 17 23:23:21.532157 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:23:21.532099    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/afb89dd1-3bed-4626-981b-fdc7f7d1f3de-kserve-provision-location\") pod \"isvc-canary-stop-predictor-596476db66-ckjrl\" (UID: \"afb89dd1-3bed-4626-981b-fdc7f7d1f3de\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-596476db66-ckjrl"
Sep 17 23:23:21.604387 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:23:21.604365    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:23:21.632729 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:23:21.632701    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/afb89dd1-3bed-4626-981b-fdc7f7d1f3de-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-596476db66-ckjrl\" (UID: \"afb89dd1-3bed-4626-981b-fdc7f7d1f3de\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-596476db66-ckjrl"
Sep 17 23:23:21.632822 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:23:21.632746    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r7j2m\" (UniqueName: \"kubernetes.io/projected/afb89dd1-3bed-4626-981b-fdc7f7d1f3de-kube-api-access-r7j2m\") pod \"isvc-canary-stop-predictor-596476db66-ckjrl\" (UID: \"afb89dd1-3bed-4626-981b-fdc7f7d1f3de\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-596476db66-ckjrl"
Sep 17 23:23:21.632822 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:23:21.632801    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/afb89dd1-3bed-4626-981b-fdc7f7d1f3de-proxy-tls\") pod \"isvc-canary-stop-predictor-596476db66-ckjrl\" (UID: \"afb89dd1-3bed-4626-981b-fdc7f7d1f3de\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-596476db66-ckjrl"
Sep 17 23:23:21.632925 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:23:21.632832    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/afb89dd1-3bed-4626-981b-fdc7f7d1f3de-kserve-provision-location\") pod \"isvc-canary-stop-predictor-596476db66-ckjrl\" (UID: \"afb89dd1-3bed-4626-981b-fdc7f7d1f3de\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-596476db66-ckjrl"
Sep 17 23:23:21.632925 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:23:21.632883    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/afb89dd1-3bed-4626-981b-fdc7f7d1f3de-cabundle-cert\") pod \"isvc-canary-stop-predictor-596476db66-ckjrl\" (UID: \"afb89dd1-3bed-4626-981b-fdc7f7d1f3de\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-596476db66-ckjrl"
Sep 17 23:23:21.633335 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:23:21.633318    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/afb89dd1-3bed-4626-981b-fdc7f7d1f3de-kserve-provision-location\") pod \"isvc-canary-stop-predictor-596476db66-ckjrl\" (UID: \"afb89dd1-3bed-4626-981b-fdc7f7d1f3de\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-596476db66-ckjrl"
Sep 17 23:23:21.633688 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:23:21.633633    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/afb89dd1-3bed-4626-981b-fdc7f7d1f3de-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-596476db66-ckjrl\" (UID: \"afb89dd1-3bed-4626-981b-fdc7f7d1f3de\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-596476db66-ckjrl"
Sep 17 23:23:21.633688 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:23:21.633654    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/afb89dd1-3bed-4626-981b-fdc7f7d1f3de-cabundle-cert\") pod \"isvc-canary-stop-predictor-596476db66-ckjrl\" (UID: \"afb89dd1-3bed-4626-981b-fdc7f7d1f3de\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-596476db66-ckjrl"
Sep 17 23:23:21.635471 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:23:21.635450    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/afb89dd1-3bed-4626-981b-fdc7f7d1f3de-proxy-tls\") pod \"isvc-canary-stop-predictor-596476db66-ckjrl\" (UID: \"afb89dd1-3bed-4626-981b-fdc7f7d1f3de\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-596476db66-ckjrl"
Sep 17 23:23:21.640690 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:23:21.640668    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r7j2m\" (UniqueName: \"kubernetes.io/projected/afb89dd1-3bed-4626-981b-fdc7f7d1f3de-kube-api-access-r7j2m\") pod \"isvc-canary-stop-predictor-596476db66-ckjrl\" (UID: \"afb89dd1-3bed-4626-981b-fdc7f7d1f3de\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-596476db66-ckjrl"
Sep 17 23:23:21.747156 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:23:21.747099    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-596476db66-ckjrl"
Sep 17 23:23:22.026775 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:23:22.026752    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-596476db66-ckjrl"]
Sep 17 23:23:22.028803 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:23:22.028781    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podafb89dd1_3bed_4626_981b_fdc7f7d1f3de.slice/crio-4e7a56d2c5bc2fb50e276b57a7e2e624b91b897a1a957961f86ca597dca7e777 WatchSource:0}: Error finding container 4e7a56d2c5bc2fb50e276b57a7e2e624b91b897a1a957961f86ca597dca7e777: Status 404 returned error can't find the container with id 4e7a56d2c5bc2fb50e276b57a7e2e624b91b897a1a957961f86ca597dca7e777
Sep 17 23:23:22.115793 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:23:22.115762    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-596476db66-ckjrl" event={"ID":"afb89dd1-3bed-4626-981b-fdc7f7d1f3de","Type":"ContainerStarted","Data":"6e4822525fef558ba1cdcefe4380fab19a0a1199f002dcf81307d66ae0381f66"}
Sep 17 23:23:22.115901 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:23:22.115824    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-596476db66-ckjrl" event={"ID":"afb89dd1-3bed-4626-981b-fdc7f7d1f3de","Type":"ContainerStarted","Data":"4e7a56d2c5bc2fb50e276b57a7e2e624b91b897a1a957961f86ca597dca7e777"}
Sep 17 23:23:22.763153 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:23:22.763119    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="afdeceb7-80fb-4e2e-8493-3c113ee34e82" path="/var/lib/kubelet/pods/afdeceb7-80fb-4e2e-8493-3c113ee34e82/volumes"
Sep 17 23:23:26.132957 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:23:26.132881    2584 generic.go:356] "Generic (PLEG): container finished" podID="afb89dd1-3bed-4626-981b-fdc7f7d1f3de" containerID="6e4822525fef558ba1cdcefe4380fab19a0a1199f002dcf81307d66ae0381f66" exitCode=0
Sep 17 23:23:26.133278 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:23:26.132956    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-596476db66-ckjrl" event={"ID":"afb89dd1-3bed-4626-981b-fdc7f7d1f3de","Type":"ContainerDied","Data":"6e4822525fef558ba1cdcefe4380fab19a0a1199f002dcf81307d66ae0381f66"}
Sep 17 23:23:27.140063 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:23:27.140031    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-596476db66-ckjrl" event={"ID":"afb89dd1-3bed-4626-981b-fdc7f7d1f3de","Type":"ContainerStarted","Data":"a7862ab15ab9853fe063a9533f91704aa2e5fb7fbc6e963ad23ec84e9898179e"}
Sep 17 23:23:27.140421 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:23:27.140092    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-596476db66-ckjrl" event={"ID":"afb89dd1-3bed-4626-981b-fdc7f7d1f3de","Type":"ContainerStarted","Data":"0ca410ad58dd596f3967812647d5f0c79697616088defecd2781df4d4c980890"}
Sep 17 23:23:27.140421 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:23:27.140355    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-596476db66-ckjrl"
Sep 17 23:23:27.140509 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:23:27.140489    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-596476db66-ckjrl"
Sep 17 23:23:27.141710 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:23:27.141689    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-596476db66-ckjrl" podUID="afb89dd1-3bed-4626-981b-fdc7f7d1f3de" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.22:8080: connect: connection refused"
Sep 17 23:23:27.160092 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:23:27.160041    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-596476db66-ckjrl" podStartSLOduration=2.38573538 podStartE2EDuration="6.160027471s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.774292091s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:23:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:23:27.159042062 +0000 UTC m=+790.524701979" watchObservedRunningTime="2026-09-17 23:23:27.160027471 +0000 UTC m=+790.525687379"
Sep 17 23:23:28.144021 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:23:28.143971    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-596476db66-ckjrl" podUID="afb89dd1-3bed-4626-981b-fdc7f7d1f3de" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.22:8080: connect: connection refused"
Sep 17 23:23:31.280752 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:23:31.280717    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/model-chainer-raw-7b7df-5957dcc4dd-rlrd4"]
Sep 17 23:23:31.284242 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:23:31.284225    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-raw-7b7df-5957dcc4dd-rlrd4"
Sep 17 23:23:31.285615 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:23:31.285585    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:23:31.287290 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:23:31.287267    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"model-chainer-raw-7b7df-kube-rbac-proxy-sar-config\""
Sep 17 23:23:31.287290 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:23:31.287267    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"model-chainer-raw-7b7df-serving-cert\""
Sep 17 23:23:31.293748 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:23:31.293722    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/model-chainer-raw-7b7df-5957dcc4dd-rlrd4"]
Sep 17 23:23:31.404775 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:23:31.404745    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fe257d69-6de9-4adc-a955-da2e32dc012d-proxy-tls\") pod \"model-chainer-raw-7b7df-5957dcc4dd-rlrd4\" (UID: \"fe257d69-6de9-4adc-a955-da2e32dc012d\") " pod="kserve-ci-e2e-test/model-chainer-raw-7b7df-5957dcc4dd-rlrd4"
Sep 17 23:23:31.404889 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:23:31.404804    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fe257d69-6de9-4adc-a955-da2e32dc012d-openshift-service-ca-bundle\") pod \"model-chainer-raw-7b7df-5957dcc4dd-rlrd4\" (UID: \"fe257d69-6de9-4adc-a955-da2e32dc012d\") " pod="kserve-ci-e2e-test/model-chainer-raw-7b7df-5957dcc4dd-rlrd4"
Sep 17 23:23:31.440898 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:23:31.440870    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:23:31.505938 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:23:31.505915    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fe257d69-6de9-4adc-a955-da2e32dc012d-openshift-service-ca-bundle\") pod \"model-chainer-raw-7b7df-5957dcc4dd-rlrd4\" (UID: \"fe257d69-6de9-4adc-a955-da2e32dc012d\") " pod="kserve-ci-e2e-test/model-chainer-raw-7b7df-5957dcc4dd-rlrd4"
Sep 17 23:23:31.506020 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:23:31.505961    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fe257d69-6de9-4adc-a955-da2e32dc012d-proxy-tls\") pod \"model-chainer-raw-7b7df-5957dcc4dd-rlrd4\" (UID: \"fe257d69-6de9-4adc-a955-da2e32dc012d\") " pod="kserve-ci-e2e-test/model-chainer-raw-7b7df-5957dcc4dd-rlrd4"
Sep 17 23:23:31.506573 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:23:31.506552    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fe257d69-6de9-4adc-a955-da2e32dc012d-openshift-service-ca-bundle\") pod \"model-chainer-raw-7b7df-5957dcc4dd-rlrd4\" (UID: \"fe257d69-6de9-4adc-a955-da2e32dc012d\") " pod="kserve-ci-e2e-test/model-chainer-raw-7b7df-5957dcc4dd-rlrd4"
Sep 17 23:23:31.508389 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:23:31.508372    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fe257d69-6de9-4adc-a955-da2e32dc012d-proxy-tls\") pod \"model-chainer-raw-7b7df-5957dcc4dd-rlrd4\" (UID: \"fe257d69-6de9-4adc-a955-da2e32dc012d\") " pod="kserve-ci-e2e-test/model-chainer-raw-7b7df-5957dcc4dd-rlrd4"
Sep 17 23:23:31.594566 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:23:31.594510    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-raw-7b7df-5957dcc4dd-rlrd4"
Sep 17 23:23:31.862521 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:23:31.862488    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/model-chainer-raw-7b7df-5957dcc4dd-rlrd4"]
Sep 17 23:23:31.865103 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:23:31.865075    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe257d69_6de9_4adc_a955_da2e32dc012d.slice/crio-69f9353762f9799dd44cb0ddbb51296d093ff521a703ad3fbfe86abad54e263d WatchSource:0}: Error finding container 69f9353762f9799dd44cb0ddbb51296d093ff521a703ad3fbfe86abad54e263d: Status 404 returned error can't find the container with id 69f9353762f9799dd44cb0ddbb51296d093ff521a703ad3fbfe86abad54e263d
Sep 17 23:23:31.866919 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:23:31.866903    2584 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 23:23:32.161321 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:23:32.161248    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-raw-7b7df-5957dcc4dd-rlrd4" event={"ID":"fe257d69-6de9-4adc-a955-da2e32dc012d","Type":"ContainerStarted","Data":"69f9353762f9799dd44cb0ddbb51296d093ff521a703ad3fbfe86abad54e263d"}
Sep 17 23:23:33.148924 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:23:33.148891    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-596476db66-ckjrl"
Sep 17 23:23:33.149549 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:23:33.149512    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-596476db66-ckjrl" podUID="afb89dd1-3bed-4626-981b-fdc7f7d1f3de" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.22:8080: connect: connection refused"
Sep 17 23:23:34.172458 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:23:34.172371    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-raw-7b7df-5957dcc4dd-rlrd4" event={"ID":"fe257d69-6de9-4adc-a955-da2e32dc012d","Type":"ContainerStarted","Data":"18457c5065ab3b5e8dcabe70cc261f44fa3937a429565558c55746022facfeb6"}
Sep 17 23:23:34.172821 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:23:34.172473    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/model-chainer-raw-7b7df-5957dcc4dd-rlrd4"
Sep 17 23:23:34.189786 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:23:34.189744    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/model-chainer-raw-7b7df-5957dcc4dd-rlrd4" podStartSLOduration=1.182003852 podStartE2EDuration="3.189729751s" totalImagesPullingTime="2.007725899s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:23:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:23:34.188133818 +0000 UTC m=+797.553793736" watchObservedRunningTime="2026-09-17 23:23:34.189729751 +0000 UTC m=+797.555389668"
Sep 17 23:23:40.182772 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:23:40.182741    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/model-chainer-raw-7b7df-5957dcc4dd-rlrd4"
Sep 17 23:23:41.335060 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:23:41.335030    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/model-chainer-raw-7b7df-5957dcc4dd-rlrd4"]
Sep 17 23:23:41.335419 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:23:41.335222    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/model-chainer-raw-7b7df-5957dcc4dd-rlrd4" podUID="fe257d69-6de9-4adc-a955-da2e32dc012d" containerName="model-chainer-raw-7b7df" containerID="cri-o://18457c5065ab3b5e8dcabe70cc261f44fa3937a429565558c55746022facfeb6" gracePeriod=30
Sep 17 23:23:41.356044 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:23:41.354114    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:23:41.470773 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:23:41.470726    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:23:41.566438 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:23:41.566393    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:23:41.598921 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:23:41.598854    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-d5772-predictor-54794bf88c-ptxqp"]
Sep 17 23:23:41.602532 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:23:41.602513    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-d5772-predictor-54794bf88c-ptxqp"
Sep 17 23:23:41.604544 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:23:41.604521    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-hpa-d5772-predictor-serving-cert\""
Sep 17 23:23:41.604646 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:23:41.604629    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-hpa-d5772-kube-rbac-proxy-sar-config\""
Sep 17 23:23:41.615578 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:23:41.615557    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-d5772-predictor-54794bf88c-ptxqp"]
Sep 17 23:23:41.642157 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:23:41.642125    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:23:41.678450 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:23:41.678422    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4870289b-2f1d-468b-a0ee-c860b460b581-cabundle-cert\") pod \"isvc-sklearn-graph-raw-hpa-d5772-predictor-54794bf88c-ptxqp\" (UID: \"4870289b-2f1d-468b-a0ee-c860b460b581\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-d5772-predictor-54794bf88c-ptxqp"
Sep 17 23:23:41.678564 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:23:41.678465    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4870289b-2f1d-468b-a0ee-c860b460b581-proxy-tls\") pod \"isvc-sklearn-graph-raw-hpa-d5772-predictor-54794bf88c-ptxqp\" (UID: \"4870289b-2f1d-468b-a0ee-c860b460b581\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-d5772-predictor-54794bf88c-ptxqp"
Sep 17 23:23:41.678564 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:23:41.678491    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cf2jv\" (UniqueName: \"kubernetes.io/projected/4870289b-2f1d-468b-a0ee-c860b460b581-kube-api-access-cf2jv\") pod \"isvc-sklearn-graph-raw-hpa-d5772-predictor-54794bf88c-ptxqp\" (UID: \"4870289b-2f1d-468b-a0ee-c860b460b581\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-d5772-predictor-54794bf88c-ptxqp"
Sep 17 23:23:41.678654 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:23:41.678564    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4870289b-2f1d-468b-a0ee-c860b460b581-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-hpa-d5772-predictor-54794bf88c-ptxqp\" (UID: \"4870289b-2f1d-468b-a0ee-c860b460b581\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-d5772-predictor-54794bf88c-ptxqp"
Sep 17 23:23:41.678654 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:23:41.678596    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4870289b-2f1d-468b-a0ee-c860b460b581-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-hpa-d5772-predictor-54794bf88c-ptxqp\" (UID: \"4870289b-2f1d-468b-a0ee-c860b460b581\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-d5772-predictor-54794bf88c-ptxqp"
Sep 17 23:23:41.696386 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:23:41.696354    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-d5772-predictor-56c57b5777-jrznx"]
Sep 17 23:23:41.699982 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:23:41.699966    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-d5772-predictor-56c57b5777-jrznx"
Sep 17 23:23:41.702132 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:23:41.702115    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-raw-hpa-d5772-predictor-serving-cert\""
Sep 17 23:23:41.702225 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:23:41.702154    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-raw-hpa-d5772-kube-rbac-proxy-sar-config\""
Sep 17 23:23:41.710254 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:23:41.710233    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-d5772-predictor-56c57b5777-jrznx"]
Sep 17 23:23:41.735448 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:23:41.735421    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:23:41.779825 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:23:41.779800    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c490c451-0744-42f3-a5d8-89fe843e7530-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-hpa-d5772-predictor-56c57b5777-jrznx\" (UID: \"c490c451-0744-42f3-a5d8-89fe843e7530\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-d5772-predictor-56c57b5777-jrznx"
Sep 17 23:23:41.779903 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:23:41.779840    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4870289b-2f1d-468b-a0ee-c860b460b581-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-hpa-d5772-predictor-54794bf88c-ptxqp\" (UID: \"4870289b-2f1d-468b-a0ee-c860b460b581\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-d5772-predictor-54794bf88c-ptxqp"
Sep 17 23:23:41.779903 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:23:41.779860    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4870289b-2f1d-468b-a0ee-c860b460b581-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-hpa-d5772-predictor-54794bf88c-ptxqp\" (UID: \"4870289b-2f1d-468b-a0ee-c860b460b581\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-d5772-predictor-54794bf88c-ptxqp"
Sep 17 23:23:41.780039 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:23:41.779943    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4870289b-2f1d-468b-a0ee-c860b460b581-cabundle-cert\") pod \"isvc-sklearn-graph-raw-hpa-d5772-predictor-54794bf88c-ptxqp\" (UID: \"4870289b-2f1d-468b-a0ee-c860b460b581\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-d5772-predictor-54794bf88c-ptxqp"
Sep 17 23:23:41.780039 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:23:41.779992    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4870289b-2f1d-468b-a0ee-c860b460b581-proxy-tls\") pod \"isvc-sklearn-graph-raw-hpa-d5772-predictor-54794bf88c-ptxqp\" (UID: \"4870289b-2f1d-468b-a0ee-c860b460b581\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-d5772-predictor-54794bf88c-ptxqp"
Sep 17 23:23:41.780154 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:23:41.780043    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c490c451-0744-42f3-a5d8-89fe843e7530-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-hpa-d5772-predictor-56c57b5777-jrznx\" (UID: \"c490c451-0744-42f3-a5d8-89fe843e7530\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-d5772-predictor-56c57b5777-jrznx"
Sep 17 23:23:41.780154 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:23:41.780093    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cf2jv\" (UniqueName: \"kubernetes.io/projected/4870289b-2f1d-468b-a0ee-c860b460b581-kube-api-access-cf2jv\") pod \"isvc-sklearn-graph-raw-hpa-d5772-predictor-54794bf88c-ptxqp\" (UID: \"4870289b-2f1d-468b-a0ee-c860b460b581\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-d5772-predictor-54794bf88c-ptxqp"
Sep 17 23:23:41.780154 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:23:41.780111    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sbctp\" (UniqueName: \"kubernetes.io/projected/c490c451-0744-42f3-a5d8-89fe843e7530-kube-api-access-sbctp\") pod \"isvc-xgboost-graph-raw-hpa-d5772-predictor-56c57b5777-jrznx\" (UID: \"c490c451-0744-42f3-a5d8-89fe843e7530\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-d5772-predictor-56c57b5777-jrznx"
Sep 17 23:23:41.780154 ip-10-0-142-4 kubenswrapper[2584]: E0917 23:23:41.780122    2584 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-d5772-predictor-serving-cert: secret "isvc-sklearn-graph-raw-hpa-d5772-predictor-serving-cert" not found
Sep 17 23:23:41.780154 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:23:41.780132    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c490c451-0744-42f3-a5d8-89fe843e7530-cabundle-cert\") pod \"isvc-xgboost-graph-raw-hpa-d5772-predictor-56c57b5777-jrznx\" (UID: \"c490c451-0744-42f3-a5d8-89fe843e7530\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-d5772-predictor-56c57b5777-jrznx"
Sep 17 23:23:41.780442 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:23:41.780164    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c490c451-0744-42f3-a5d8-89fe843e7530-proxy-tls\") pod \"isvc-xgboost-graph-raw-hpa-d5772-predictor-56c57b5777-jrznx\" (UID: \"c490c451-0744-42f3-a5d8-89fe843e7530\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-d5772-predictor-56c57b5777-jrznx"
Sep 17 23:23:41.780442 ip-10-0-142-4 kubenswrapper[2584]: E0917 23:23:41.780192    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4870289b-2f1d-468b-a0ee-c860b460b581-proxy-tls podName:4870289b-2f1d-468b-a0ee-c860b460b581 nodeName:}" failed. No retries permitted until 2026-09-17 23:23:42.280174088 +0000 UTC m=+805.645833984 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/4870289b-2f1d-468b-a0ee-c860b460b581-proxy-tls") pod "isvc-sklearn-graph-raw-hpa-d5772-predictor-54794bf88c-ptxqp" (UID: "4870289b-2f1d-468b-a0ee-c860b460b581") : secret "isvc-sklearn-graph-raw-hpa-d5772-predictor-serving-cert" not found
Sep 17 23:23:41.780442 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:23:41.780252    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4870289b-2f1d-468b-a0ee-c860b460b581-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-hpa-d5772-predictor-54794bf88c-ptxqp\" (UID: \"4870289b-2f1d-468b-a0ee-c860b460b581\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-d5772-predictor-54794bf88c-ptxqp"
Sep 17 23:23:41.780624 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:23:41.780527    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4870289b-2f1d-468b-a0ee-c860b460b581-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-hpa-d5772-predictor-54794bf88c-ptxqp\" (UID: \"4870289b-2f1d-468b-a0ee-c860b460b581\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-d5772-predictor-54794bf88c-ptxqp"
Sep 17 23:23:41.780675 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:23:41.780658    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4870289b-2f1d-468b-a0ee-c860b460b581-cabundle-cert\") pod \"isvc-sklearn-graph-raw-hpa-d5772-predictor-54794bf88c-ptxqp\" (UID: \"4870289b-2f1d-468b-a0ee-c860b460b581\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-d5772-predictor-54794bf88c-ptxqp"
Sep 17 23:23:41.789077 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:23:41.789059    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cf2jv\" (UniqueName: \"kubernetes.io/projected/4870289b-2f1d-468b-a0ee-c860b460b581-kube-api-access-cf2jv\") pod \"isvc-sklearn-graph-raw-hpa-d5772-predictor-54794bf88c-ptxqp\" (UID: \"4870289b-2f1d-468b-a0ee-c860b460b581\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-d5772-predictor-54794bf88c-ptxqp"
Sep 17 23:23:41.839187 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:23:41.839164    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:23:41.880672 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:23:41.880618    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c490c451-0744-42f3-a5d8-89fe843e7530-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-hpa-d5772-predictor-56c57b5777-jrznx\" (UID: \"c490c451-0744-42f3-a5d8-89fe843e7530\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-d5772-predictor-56c57b5777-jrznx"
Sep 17 23:23:41.880672 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:23:41.880651    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sbctp\" (UniqueName: \"kubernetes.io/projected/c490c451-0744-42f3-a5d8-89fe843e7530-kube-api-access-sbctp\") pod \"isvc-xgboost-graph-raw-hpa-d5772-predictor-56c57b5777-jrznx\" (UID: \"c490c451-0744-42f3-a5d8-89fe843e7530\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-d5772-predictor-56c57b5777-jrznx"
Sep 17 23:23:41.880672 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:23:41.880669    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c490c451-0744-42f3-a5d8-89fe843e7530-cabundle-cert\") pod \"isvc-xgboost-graph-raw-hpa-d5772-predictor-56c57b5777-jrznx\" (UID: \"c490c451-0744-42f3-a5d8-89fe843e7530\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-d5772-predictor-56c57b5777-jrznx"
Sep 17 23:23:41.880838 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:23:41.880687    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c490c451-0744-42f3-a5d8-89fe843e7530-proxy-tls\") pod \"isvc-xgboost-graph-raw-hpa-d5772-predictor-56c57b5777-jrznx\" (UID: \"c490c451-0744-42f3-a5d8-89fe843e7530\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-d5772-predictor-56c57b5777-jrznx"
Sep 17 23:23:41.880838 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:23:41.880704    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c490c451-0744-42f3-a5d8-89fe843e7530-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-hpa-d5772-predictor-56c57b5777-jrznx\" (UID: \"c490c451-0744-42f3-a5d8-89fe843e7530\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-d5772-predictor-56c57b5777-jrznx"
Sep 17 23:23:41.881059 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:23:41.881041    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c490c451-0744-42f3-a5d8-89fe843e7530-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-hpa-d5772-predictor-56c57b5777-jrznx\" (UID: \"c490c451-0744-42f3-a5d8-89fe843e7530\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-d5772-predictor-56c57b5777-jrznx"
Sep 17 23:23:41.881315 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:23:41.881298    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c490c451-0744-42f3-a5d8-89fe843e7530-cabundle-cert\") pod \"isvc-xgboost-graph-raw-hpa-d5772-predictor-56c57b5777-jrznx\" (UID: \"c490c451-0744-42f3-a5d8-89fe843e7530\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-d5772-predictor-56c57b5777-jrznx"
Sep 17 23:23:41.881399 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:23:41.881383    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c490c451-0744-42f3-a5d8-89fe843e7530-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-hpa-d5772-predictor-56c57b5777-jrznx\" (UID: \"c490c451-0744-42f3-a5d8-89fe843e7530\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-d5772-predictor-56c57b5777-jrznx"
Sep 17 23:23:41.883321 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:23:41.883303    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c490c451-0744-42f3-a5d8-89fe843e7530-proxy-tls\") pod \"isvc-xgboost-graph-raw-hpa-d5772-predictor-56c57b5777-jrznx\" (UID: \"c490c451-0744-42f3-a5d8-89fe843e7530\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-d5772-predictor-56c57b5777-jrznx"
Sep 17 23:23:41.889499 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:23:41.889481    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sbctp\" (UniqueName: \"kubernetes.io/projected/c490c451-0744-42f3-a5d8-89fe843e7530-kube-api-access-sbctp\") pod \"isvc-xgboost-graph-raw-hpa-d5772-predictor-56c57b5777-jrznx\" (UID: \"c490c451-0744-42f3-a5d8-89fe843e7530\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-d5772-predictor-56c57b5777-jrznx"
Sep 17 23:23:41.940442 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:23:41.940415    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:23:42.010616 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:23:42.010594    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-d5772-predictor-56c57b5777-jrznx"
Sep 17 23:23:42.283740 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:23:42.283716    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-d5772-predictor-56c57b5777-jrznx"]
Sep 17 23:23:42.284171 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:23:42.284149    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4870289b-2f1d-468b-a0ee-c860b460b581-proxy-tls\") pod \"isvc-sklearn-graph-raw-hpa-d5772-predictor-54794bf88c-ptxqp\" (UID: \"4870289b-2f1d-468b-a0ee-c860b460b581\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-d5772-predictor-54794bf88c-ptxqp"
Sep 17 23:23:42.285982 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:23:42.285957    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc490c451_0744_42f3_a5d8_89fe843e7530.slice/crio-83004f678aae3e54654e1a0ebd6310cdbcfbf296fe7aacbf857541785b733d38 WatchSource:0}: Error finding container 83004f678aae3e54654e1a0ebd6310cdbcfbf296fe7aacbf857541785b733d38: Status 404 returned error can't find the container with id 83004f678aae3e54654e1a0ebd6310cdbcfbf296fe7aacbf857541785b733d38
Sep 17 23:23:42.286775 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:23:42.286757    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4870289b-2f1d-468b-a0ee-c860b460b581-proxy-tls\") pod \"isvc-sklearn-graph-raw-hpa-d5772-predictor-54794bf88c-ptxqp\" (UID: \"4870289b-2f1d-468b-a0ee-c860b460b581\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-d5772-predictor-54794bf88c-ptxqp"
Sep 17 23:23:42.512187 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:23:42.512149    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-d5772-predictor-54794bf88c-ptxqp"
Sep 17 23:23:42.796329 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:23:42.796297    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-d5772-predictor-54794bf88c-ptxqp"]
Sep 17 23:23:42.800174 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:23:42.800151    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4870289b_2f1d_468b_a0ee_c860b460b581.slice/crio-e4ad8451d5d0a84cbaf3e0d1c9b40eb4abb743a80f4c9d4f387f1b9349bdb621 WatchSource:0}: Error finding container e4ad8451d5d0a84cbaf3e0d1c9b40eb4abb743a80f4c9d4f387f1b9349bdb621: Status 404 returned error can't find the container with id e4ad8451d5d0a84cbaf3e0d1c9b40eb4abb743a80f4c9d4f387f1b9349bdb621
Sep 17 23:23:43.149565 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:23:43.149474    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-596476db66-ckjrl" podUID="afb89dd1-3bed-4626-981b-fdc7f7d1f3de" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.22:8080: connect: connection refused"
Sep 17 23:23:43.212713 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:23:43.212672    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-d5772-predictor-56c57b5777-jrznx" event={"ID":"c490c451-0744-42f3-a5d8-89fe843e7530","Type":"ContainerStarted","Data":"86fd5096362a6af8899b59a3fc3ee00f2d23ba3dd648f8b80f2878c53b24e01e"}
Sep 17 23:23:43.212837 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:23:43.212750    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-d5772-predictor-56c57b5777-jrznx" event={"ID":"c490c451-0744-42f3-a5d8-89fe843e7530","Type":"ContainerStarted","Data":"83004f678aae3e54654e1a0ebd6310cdbcfbf296fe7aacbf857541785b733d38"}
Sep 17 23:23:43.214420 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:23:43.214392    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-d5772-predictor-54794bf88c-ptxqp" event={"ID":"4870289b-2f1d-468b-a0ee-c860b460b581","Type":"ContainerStarted","Data":"76d931b06d543340959f1bfcf7bb893d73864e78e7e8b12392afa7e633723095"}
Sep 17 23:23:43.214536 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:23:43.214458    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-d5772-predictor-54794bf88c-ptxqp" event={"ID":"4870289b-2f1d-468b-a0ee-c860b460b581","Type":"ContainerStarted","Data":"e4ad8451d5d0a84cbaf3e0d1c9b40eb4abb743a80f4c9d4f387f1b9349bdb621"}
Sep 17 23:23:45.181137 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:23:45.181096    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-raw-7b7df-5957dcc4dd-rlrd4" podUID="fe257d69-6de9-4adc-a955-da2e32dc012d" containerName="model-chainer-raw-7b7df" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 23:23:46.235450 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:23:46.235418    2584 generic.go:356] "Generic (PLEG): container finished" podID="c490c451-0744-42f3-a5d8-89fe843e7530" containerID="86fd5096362a6af8899b59a3fc3ee00f2d23ba3dd648f8b80f2878c53b24e01e" exitCode=0
Sep 17 23:23:46.235864 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:23:46.235495    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-d5772-predictor-56c57b5777-jrznx" event={"ID":"c490c451-0744-42f3-a5d8-89fe843e7530","Type":"ContainerDied","Data":"86fd5096362a6af8899b59a3fc3ee00f2d23ba3dd648f8b80f2878c53b24e01e"}
Sep 17 23:23:47.243637 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:23:47.243524    2584 generic.go:356] "Generic (PLEG): container finished" podID="4870289b-2f1d-468b-a0ee-c860b460b581" containerID="76d931b06d543340959f1bfcf7bb893d73864e78e7e8b12392afa7e633723095" exitCode=0
Sep 17 23:23:47.243637 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:23:47.243597    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-d5772-predictor-54794bf88c-ptxqp" event={"ID":"4870289b-2f1d-468b-a0ee-c860b460b581","Type":"ContainerDied","Data":"76d931b06d543340959f1bfcf7bb893d73864e78e7e8b12392afa7e633723095"}
Sep 17 23:23:49.257741 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:23:49.257707    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-d5772-predictor-54794bf88c-ptxqp" event={"ID":"4870289b-2f1d-468b-a0ee-c860b460b581","Type":"ContainerStarted","Data":"b240b00217329796d85e516a8928c38007e0bd9872c0b04a9326ae079f24d093"}
Sep 17 23:23:49.258237 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:23:49.257772    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-d5772-predictor-54794bf88c-ptxqp" event={"ID":"4870289b-2f1d-468b-a0ee-c860b460b581","Type":"ContainerStarted","Data":"6c45bf1c18d69351550db4e8a75bfe0f0b17b44a957317117312a0c50bd297e4"}
Sep 17 23:23:49.258237 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:23:49.257981    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-d5772-predictor-54794bf88c-ptxqp"
Sep 17 23:23:49.295914 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:23:49.295869    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-d5772-predictor-54794bf88c-ptxqp" podStartSLOduration=4.262396633 podStartE2EDuration="8.295855263s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.03345863s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:23:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:23:49.295574904 +0000 UTC m=+812.661234824" watchObservedRunningTime="2026-09-17 23:23:49.295855263 +0000 UTC m=+812.661515180"
Sep 17 23:23:50.181022 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:23:50.180960    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-raw-7b7df-5957dcc4dd-rlrd4" podUID="fe257d69-6de9-4adc-a955-da2e32dc012d" containerName="model-chainer-raw-7b7df" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 23:23:50.263280 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:23:50.263232    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-d5772-predictor-54794bf88c-ptxqp"
Sep 17 23:23:50.264755 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:23:50.264718    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-d5772-predictor-54794bf88c-ptxqp" podUID="4870289b-2f1d-468b-a0ee-c860b460b581" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:8080: connect: connection refused"
Sep 17 23:23:51.269295 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:23:51.269253    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-d5772-predictor-54794bf88c-ptxqp" podUID="4870289b-2f1d-468b-a0ee-c860b460b581" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:8080: connect: connection refused"
Sep 17 23:23:52.154777 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:23:52.154734    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:23:52.254787 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:23:52.254751    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:23:53.149627 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:23:53.149584    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-596476db66-ckjrl" podUID="afb89dd1-3bed-4626-981b-fdc7f7d1f3de" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.22:8080: connect: connection refused"
Sep 17 23:23:55.182273 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:23:55.182234    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-raw-7b7df-5957dcc4dd-rlrd4" podUID="fe257d69-6de9-4adc-a955-da2e32dc012d" containerName="model-chainer-raw-7b7df" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 23:23:55.182709 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:23:55.182342    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/model-chainer-raw-7b7df-5957dcc4dd-rlrd4"
Sep 17 23:23:56.278366 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:23:56.278331    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-d5772-predictor-54794bf88c-ptxqp"
Sep 17 23:23:56.279096 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:23:56.278999    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-d5772-predictor-54794bf88c-ptxqp" podUID="4870289b-2f1d-468b-a0ee-c860b460b581" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:8080: connect: connection refused"
Sep 17 23:24:00.181512 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:24:00.181473    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-raw-7b7df-5957dcc4dd-rlrd4" podUID="fe257d69-6de9-4adc-a955-da2e32dc012d" containerName="model-chainer-raw-7b7df" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 23:24:03.150361 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:24:03.150320    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-596476db66-ckjrl" podUID="afb89dd1-3bed-4626-981b-fdc7f7d1f3de" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.22:8080: connect: connection refused"
Sep 17 23:24:05.181316 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:24:05.181280    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-raw-7b7df-5957dcc4dd-rlrd4" podUID="fe257d69-6de9-4adc-a955-da2e32dc012d" containerName="model-chainer-raw-7b7df" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 23:24:06.279604 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:24:06.279565    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-d5772-predictor-54794bf88c-ptxqp" podUID="4870289b-2f1d-468b-a0ee-c860b460b581" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:8080: connect: connection refused"
Sep 17 23:24:06.341676 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:24:06.341643    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-d5772-predictor-56c57b5777-jrznx" event={"ID":"c490c451-0744-42f3-a5d8-89fe843e7530","Type":"ContainerStarted","Data":"934f451cf6acb5864e1449a0b7864aa9066dcd491c70e9c7b164049f781645c9"}
Sep 17 23:24:06.341832 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:24:06.341739    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-d5772-predictor-56c57b5777-jrznx" event={"ID":"c490c451-0744-42f3-a5d8-89fe843e7530","Type":"ContainerStarted","Data":"2041d9ebb58c79788e96c90edd9738be9c3576d5bd8111f48602dcb04219fe65"}
Sep 17 23:24:06.341952 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:24:06.341927    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-d5772-predictor-56c57b5777-jrznx"
Sep 17 23:24:06.363095 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:24:06.363041    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-d5772-predictor-56c57b5777-jrznx" podStartSLOduration=1.726848646 podStartE2EDuration="25.363025376s" totalImagesPullingTime="19.883102536s" totalInitContainerRuntime="3.753074194s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:23:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:24:06.36134762 +0000 UTC m=+829.727007558" watchObservedRunningTime="2026-09-17 23:24:06.363025376 +0000 UTC m=+829.728685292"
Sep 17 23:24:07.346488 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:24:07.346459    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-d5772-predictor-56c57b5777-jrznx"
Sep 17 23:24:07.347650 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:24:07.347625    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-d5772-predictor-56c57b5777-jrznx" podUID="c490c451-0744-42f3-a5d8-89fe843e7530" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
Sep 17 23:24:08.351337 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:24:08.351302    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-d5772-predictor-56c57b5777-jrznx" podUID="c490c451-0744-42f3-a5d8-89fe843e7530" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
Sep 17 23:24:10.181404 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:24:10.181365    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-raw-7b7df-5957dcc4dd-rlrd4" podUID="fe257d69-6de9-4adc-a955-da2e32dc012d" containerName="model-chainer-raw-7b7df" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 23:24:11.367787 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:24:11.367754    2584 generic.go:356] "Generic (PLEG): container finished" podID="fe257d69-6de9-4adc-a955-da2e32dc012d" containerID="18457c5065ab3b5e8dcabe70cc261f44fa3937a429565558c55746022facfeb6" exitCode=0
Sep 17 23:24:11.368200 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:24:11.367804    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-raw-7b7df-5957dcc4dd-rlrd4" event={"ID":"fe257d69-6de9-4adc-a955-da2e32dc012d","Type":"ContainerDied","Data":"18457c5065ab3b5e8dcabe70cc261f44fa3937a429565558c55746022facfeb6"}
Sep 17 23:24:11.477622 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:24:11.477600    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-raw-7b7df-5957dcc4dd-rlrd4"
Sep 17 23:24:11.538241 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:24:11.538212    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fe257d69-6de9-4adc-a955-da2e32dc012d-openshift-service-ca-bundle\") pod \"fe257d69-6de9-4adc-a955-da2e32dc012d\" (UID: \"fe257d69-6de9-4adc-a955-da2e32dc012d\") "
Sep 17 23:24:11.538417 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:24:11.538274    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fe257d69-6de9-4adc-a955-da2e32dc012d-proxy-tls\") pod \"fe257d69-6de9-4adc-a955-da2e32dc012d\" (UID: \"fe257d69-6de9-4adc-a955-da2e32dc012d\") "
Sep 17 23:24:11.538616 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:24:11.538591    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe257d69-6de9-4adc-a955-da2e32dc012d-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "fe257d69-6de9-4adc-a955-da2e32dc012d" (UID: "fe257d69-6de9-4adc-a955-da2e32dc012d"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:24:11.540430 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:24:11.540397    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fe257d69-6de9-4adc-a955-da2e32dc012d-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "fe257d69-6de9-4adc-a955-da2e32dc012d" (UID: "fe257d69-6de9-4adc-a955-da2e32dc012d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:24:11.639506 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:24:11.639444    2584 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fe257d69-6de9-4adc-a955-da2e32dc012d-proxy-tls\") on node \"ip-10-0-142-4.ec2.internal\" DevicePath \"\""
Sep 17 23:24:11.639506 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:24:11.639468    2584 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fe257d69-6de9-4adc-a955-da2e32dc012d-openshift-service-ca-bundle\") on node \"ip-10-0-142-4.ec2.internal\" DevicePath \"\""
Sep 17 23:24:12.373628 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:24:12.373597    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-raw-7b7df-5957dcc4dd-rlrd4"
Sep 17 23:24:12.374102 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:24:12.373624    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-raw-7b7df-5957dcc4dd-rlrd4" event={"ID":"fe257d69-6de9-4adc-a955-da2e32dc012d","Type":"ContainerDied","Data":"69f9353762f9799dd44cb0ddbb51296d093ff521a703ad3fbfe86abad54e263d"}
Sep 17 23:24:12.374102 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:24:12.373684    2584 scope.go:117] "RemoveContainer" containerID="18457c5065ab3b5e8dcabe70cc261f44fa3937a429565558c55746022facfeb6"
Sep 17 23:24:12.393234 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:24:12.393209    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/model-chainer-raw-7b7df-5957dcc4dd-rlrd4"]
Sep 17 23:24:12.398335 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:24:12.398310    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/model-chainer-raw-7b7df-5957dcc4dd-rlrd4"]
Sep 17 23:24:12.763750 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:24:12.763722    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fe257d69-6de9-4adc-a955-da2e32dc012d" path="/var/lib/kubelet/pods/fe257d69-6de9-4adc-a955-da2e32dc012d/volumes"
Sep 17 23:24:13.149315 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:24:13.149229    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-596476db66-ckjrl" podUID="afb89dd1-3bed-4626-981b-fdc7f7d1f3de" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.22:8080: connect: connection refused"
Sep 17 23:24:13.355656 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:24:13.355623    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-d5772-predictor-56c57b5777-jrznx"
Sep 17 23:24:13.356145 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:24:13.356120    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-d5772-predictor-56c57b5777-jrznx" podUID="c490c451-0744-42f3-a5d8-89fe843e7530" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
Sep 17 23:24:16.279416 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:24:16.279382    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-d5772-predictor-54794bf88c-ptxqp" podUID="4870289b-2f1d-468b-a0ee-c860b460b581" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:8080: connect: connection refused"
Sep 17 23:24:23.150162 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:24:23.150121    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-596476db66-ckjrl" podUID="afb89dd1-3bed-4626-981b-fdc7f7d1f3de" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.22:8080: connect: connection refused"
Sep 17 23:24:23.356180 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:24:23.356144    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-d5772-predictor-56c57b5777-jrznx" podUID="c490c451-0744-42f3-a5d8-89fe843e7530" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
Sep 17 23:24:26.279040 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:24:26.278981    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-d5772-predictor-54794bf88c-ptxqp" podUID="4870289b-2f1d-468b-a0ee-c860b460b581" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:8080: connect: connection refused"
Sep 17 23:24:33.149362 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:24:33.149327    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-596476db66-ckjrl" podUID="afb89dd1-3bed-4626-981b-fdc7f7d1f3de" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.22:8080: connect: connection refused"
Sep 17 23:24:33.356885 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:24:33.356848    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-d5772-predictor-56c57b5777-jrznx" podUID="c490c451-0744-42f3-a5d8-89fe843e7530" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
Sep 17 23:24:36.279102 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:24:36.279067    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-d5772-predictor-54794bf88c-ptxqp" podUID="4870289b-2f1d-468b-a0ee-c860b460b581" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:8080: connect: connection refused"
Sep 17 23:24:43.150145 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:24:43.150112    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-596476db66-ckjrl"
Sep 17 23:24:43.356875 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:24:43.356834    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-d5772-predictor-56c57b5777-jrznx" podUID="c490c451-0744-42f3-a5d8-89fe843e7530" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
Sep 17 23:24:46.279781 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:24:46.279743    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-d5772-predictor-54794bf88c-ptxqp" podUID="4870289b-2f1d-468b-a0ee-c860b460b581" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:8080: connect: connection refused"
Sep 17 23:24:51.402675 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:24:51.402649    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:24:51.424775 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:24:51.424751    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-596476db66-ckjrl"]
Sep 17 23:24:51.425024 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:24:51.424975    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-596476db66-ckjrl" podUID="afb89dd1-3bed-4626-981b-fdc7f7d1f3de" containerName="kserve-container" containerID="cri-o://0ca410ad58dd596f3967812647d5f0c79697616088defecd2781df4d4c980890" gracePeriod=30
Sep 17 23:24:51.425163 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:24:51.425067    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-596476db66-ckjrl" podUID="afb89dd1-3bed-4626-981b-fdc7f7d1f3de" containerName="kube-rbac-proxy" containerID="cri-o://a7862ab15ab9853fe063a9533f91704aa2e5fb7fbc6e963ad23ec84e9898179e" gracePeriod=30
Sep 17 23:24:51.468954 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:24:51.468929    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:24:52.542962 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:24:52.542931    2584 generic.go:356] "Generic (PLEG): container finished" podID="afb89dd1-3bed-4626-981b-fdc7f7d1f3de" containerID="a7862ab15ab9853fe063a9533f91704aa2e5fb7fbc6e963ad23ec84e9898179e" exitCode=2
Sep 17 23:24:52.543341 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:24:52.543023    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-596476db66-ckjrl" event={"ID":"afb89dd1-3bed-4626-981b-fdc7f7d1f3de","Type":"ContainerDied","Data":"a7862ab15ab9853fe063a9533f91704aa2e5fb7fbc6e963ad23ec84e9898179e"}
Sep 17 23:24:53.144846 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:24:53.144801    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-596476db66-ckjrl" podUID="afb89dd1-3bed-4626-981b-fdc7f7d1f3de" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.22:8643/healthz\": dial tcp 10.132.0.22:8643: connect: connection refused"
Sep 17 23:24:53.149972 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:24:53.149942    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-596476db66-ckjrl" podUID="afb89dd1-3bed-4626-981b-fdc7f7d1f3de" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.22:8080: connect: connection refused"
Sep 17 23:24:53.356718 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:24:53.356682    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-d5772-predictor-56c57b5777-jrznx" podUID="c490c451-0744-42f3-a5d8-89fe843e7530" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
Sep 17 23:24:55.557315 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:24:55.557283    2584 generic.go:356] "Generic (PLEG): container finished" podID="afb89dd1-3bed-4626-981b-fdc7f7d1f3de" containerID="0ca410ad58dd596f3967812647d5f0c79697616088defecd2781df4d4c980890" exitCode=0
Sep 17 23:24:55.557630 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:24:55.557358    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-596476db66-ckjrl" event={"ID":"afb89dd1-3bed-4626-981b-fdc7f7d1f3de","Type":"ContainerDied","Data":"0ca410ad58dd596f3967812647d5f0c79697616088defecd2781df4d4c980890"}
Sep 17 23:24:55.672586 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:24:55.672562    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-596476db66-ckjrl"
Sep 17 23:24:55.784787 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:24:55.784707    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/afb89dd1-3bed-4626-981b-fdc7f7d1f3de-cabundle-cert\") pod \"afb89dd1-3bed-4626-981b-fdc7f7d1f3de\" (UID: \"afb89dd1-3bed-4626-981b-fdc7f7d1f3de\") "
Sep 17 23:24:55.784787 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:24:55.784766    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r7j2m\" (UniqueName: \"kubernetes.io/projected/afb89dd1-3bed-4626-981b-fdc7f7d1f3de-kube-api-access-r7j2m\") pod \"afb89dd1-3bed-4626-981b-fdc7f7d1f3de\" (UID: \"afb89dd1-3bed-4626-981b-fdc7f7d1f3de\") "
Sep 17 23:24:55.784953 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:24:55.784931    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/afb89dd1-3bed-4626-981b-fdc7f7d1f3de-kserve-provision-location\") pod \"afb89dd1-3bed-4626-981b-fdc7f7d1f3de\" (UID: \"afb89dd1-3bed-4626-981b-fdc7f7d1f3de\") "
Sep 17 23:24:55.784991 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:24:55.784969    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/afb89dd1-3bed-4626-981b-fdc7f7d1f3de-proxy-tls\") pod \"afb89dd1-3bed-4626-981b-fdc7f7d1f3de\" (UID: \"afb89dd1-3bed-4626-981b-fdc7f7d1f3de\") "
Sep 17 23:24:55.785065 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:24:55.785039    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/afb89dd1-3bed-4626-981b-fdc7f7d1f3de-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "afb89dd1-3bed-4626-981b-fdc7f7d1f3de" (UID: "afb89dd1-3bed-4626-981b-fdc7f7d1f3de"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:24:55.785120 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:24:55.785083    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/afb89dd1-3bed-4626-981b-fdc7f7d1f3de-kube-rbac-proxy-sar-config\") pod \"afb89dd1-3bed-4626-981b-fdc7f7d1f3de\" (UID: \"afb89dd1-3bed-4626-981b-fdc7f7d1f3de\") "
Sep 17 23:24:55.785273 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:24:55.785244    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afb89dd1-3bed-4626-981b-fdc7f7d1f3de-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "afb89dd1-3bed-4626-981b-fdc7f7d1f3de" (UID: "afb89dd1-3bed-4626-981b-fdc7f7d1f3de"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:24:55.785455 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:24:55.785398    2584 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/afb89dd1-3bed-4626-981b-fdc7f7d1f3de-cabundle-cert\") on node \"ip-10-0-142-4.ec2.internal\" DevicePath \"\""
Sep 17 23:24:55.785554 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:24:55.785417    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/afb89dd1-3bed-4626-981b-fdc7f7d1f3de-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "afb89dd1-3bed-4626-981b-fdc7f7d1f3de" (UID: "afb89dd1-3bed-4626-981b-fdc7f7d1f3de"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:24:55.785599 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:24:55.785554    2584 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/afb89dd1-3bed-4626-981b-fdc7f7d1f3de-kserve-provision-location\") on node \"ip-10-0-142-4.ec2.internal\" DevicePath \"\""
Sep 17 23:24:55.786972 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:24:55.786948    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/afb89dd1-3bed-4626-981b-fdc7f7d1f3de-kube-api-access-r7j2m" (OuterVolumeSpecName: "kube-api-access-r7j2m") pod "afb89dd1-3bed-4626-981b-fdc7f7d1f3de" (UID: "afb89dd1-3bed-4626-981b-fdc7f7d1f3de"). InnerVolumeSpecName "kube-api-access-r7j2m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:24:55.787121 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:24:55.787098    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/afb89dd1-3bed-4626-981b-fdc7f7d1f3de-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "afb89dd1-3bed-4626-981b-fdc7f7d1f3de" (UID: "afb89dd1-3bed-4626-981b-fdc7f7d1f3de"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:24:55.886936 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:24:55.886912    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r7j2m\" (UniqueName: \"kubernetes.io/projected/afb89dd1-3bed-4626-981b-fdc7f7d1f3de-kube-api-access-r7j2m\") on node \"ip-10-0-142-4.ec2.internal\" DevicePath \"\""
Sep 17 23:24:55.886936 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:24:55.886932    2584 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/afb89dd1-3bed-4626-981b-fdc7f7d1f3de-proxy-tls\") on node \"ip-10-0-142-4.ec2.internal\" DevicePath \"\""
Sep 17 23:24:55.887086 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:24:55.886942    2584 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/afb89dd1-3bed-4626-981b-fdc7f7d1f3de-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-4.ec2.internal\" DevicePath \"\""
Sep 17 23:24:56.279715 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:24:56.279678    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-d5772-predictor-54794bf88c-ptxqp" podUID="4870289b-2f1d-468b-a0ee-c860b460b581" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:8080: connect: connection refused"
Sep 17 23:24:56.563622 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:24:56.563555    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-596476db66-ckjrl"
Sep 17 23:24:56.563622 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:24:56.563560    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-596476db66-ckjrl" event={"ID":"afb89dd1-3bed-4626-981b-fdc7f7d1f3de","Type":"ContainerDied","Data":"4e7a56d2c5bc2fb50e276b57a7e2e624b91b897a1a957961f86ca597dca7e777"}
Sep 17 23:24:56.563622 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:24:56.563620    2584 scope.go:117] "RemoveContainer" containerID="a7862ab15ab9853fe063a9533f91704aa2e5fb7fbc6e963ad23ec84e9898179e"
Sep 17 23:24:56.572519 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:24:56.572502    2584 scope.go:117] "RemoveContainer" containerID="0ca410ad58dd596f3967812647d5f0c79697616088defecd2781df4d4c980890"
Sep 17 23:24:56.579747 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:24:56.579730    2584 scope.go:117] "RemoveContainer" containerID="6e4822525fef558ba1cdcefe4380fab19a0a1199f002dcf81307d66ae0381f66"
Sep 17 23:24:56.585993 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:24:56.585970    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-596476db66-ckjrl"]
Sep 17 23:24:56.588953 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:24:56.588926    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-596476db66-ckjrl"]
Sep 17 23:24:56.763234 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:24:56.763207    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="afb89dd1-3bed-4626-981b-fdc7f7d1f3de" path="/var/lib/kubelet/pods/afb89dd1-3bed-4626-981b-fdc7f7d1f3de/volumes"
Sep 17 23:25:01.633819 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:25:01.633749    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:25:01.772145 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:25:01.772117    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:25:03.356101 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:25:03.356066    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-d5772-predictor-56c57b5777-jrznx" podUID="c490c451-0744-42f3-a5d8-89fe843e7530" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
Sep 17 23:25:06.280157 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:25:06.280128    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-d5772-predictor-54794bf88c-ptxqp"
Sep 17 23:25:13.356574 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:25:13.356545    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-d5772-predictor-56c57b5777-jrznx"
Sep 17 23:25:21.563369 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:25:21.563333    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/model-chainer-raw-hpa-d5772-75ffc6d89c-k2m2h"]
Sep 17 23:25:21.565681 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:25:21.563655    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="afb89dd1-3bed-4626-981b-fdc7f7d1f3de" containerName="kube-rbac-proxy"
Sep 17 23:25:21.565681 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:25:21.563666    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="afb89dd1-3bed-4626-981b-fdc7f7d1f3de" containerName="kube-rbac-proxy"
Sep 17 23:25:21.565681 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:25:21.563675    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fe257d69-6de9-4adc-a955-da2e32dc012d" containerName="model-chainer-raw-7b7df"
Sep 17 23:25:21.565681 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:25:21.563681    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="fe257d69-6de9-4adc-a955-da2e32dc012d" containerName="model-chainer-raw-7b7df"
Sep 17 23:25:21.565681 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:25:21.563692    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="afb89dd1-3bed-4626-981b-fdc7f7d1f3de" containerName="storage-initializer"
Sep 17 23:25:21.565681 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:25:21.563697    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="afb89dd1-3bed-4626-981b-fdc7f7d1f3de" containerName="storage-initializer"
Sep 17 23:25:21.565681 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:25:21.563710    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="afb89dd1-3bed-4626-981b-fdc7f7d1f3de" containerName="kserve-container"
Sep 17 23:25:21.565681 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:25:21.563716    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="afb89dd1-3bed-4626-981b-fdc7f7d1f3de" containerName="kserve-container"
Sep 17 23:25:21.565681 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:25:21.563772    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="afb89dd1-3bed-4626-981b-fdc7f7d1f3de" containerName="kube-rbac-proxy"
Sep 17 23:25:21.565681 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:25:21.563779    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="fe257d69-6de9-4adc-a955-da2e32dc012d" containerName="model-chainer-raw-7b7df"
Sep 17 23:25:21.565681 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:25:21.563788    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="afb89dd1-3bed-4626-981b-fdc7f7d1f3de" containerName="kserve-container"
Sep 17 23:25:21.566563 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:25:21.566548    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-d5772-75ffc6d89c-k2m2h"
Sep 17 23:25:21.569091 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:25:21.569073    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"model-chainer-raw-hpa-d5772-serving-cert\""
Sep 17 23:25:21.569208 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:25:21.569117    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"model-chainer-raw-hpa-d5772-kube-rbac-proxy-sar-config\""
Sep 17 23:25:21.575214 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:25:21.575193    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/model-chainer-raw-hpa-d5772-75ffc6d89c-k2m2h"]
Sep 17 23:25:21.575593 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:25:21.575572    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:25:21.671450 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:25:21.671420    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5b48a893-be3e-4060-b610-3365757a2b8c-proxy-tls\") pod \"model-chainer-raw-hpa-d5772-75ffc6d89c-k2m2h\" (UID: \"5b48a893-be3e-4060-b610-3365757a2b8c\") " pod="kserve-ci-e2e-test/model-chainer-raw-hpa-d5772-75ffc6d89c-k2m2h"
Sep 17 23:25:21.671543 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:25:21.671486    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5b48a893-be3e-4060-b610-3365757a2b8c-openshift-service-ca-bundle\") pod \"model-chainer-raw-hpa-d5772-75ffc6d89c-k2m2h\" (UID: \"5b48a893-be3e-4060-b610-3365757a2b8c\") " pod="kserve-ci-e2e-test/model-chainer-raw-hpa-d5772-75ffc6d89c-k2m2h"
Sep 17 23:25:21.772684 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:25:21.772654    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5b48a893-be3e-4060-b610-3365757a2b8c-openshift-service-ca-bundle\") pod \"model-chainer-raw-hpa-d5772-75ffc6d89c-k2m2h\" (UID: \"5b48a893-be3e-4060-b610-3365757a2b8c\") " pod="kserve-ci-e2e-test/model-chainer-raw-hpa-d5772-75ffc6d89c-k2m2h"
Sep 17 23:25:21.772799 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:25:21.772709    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5b48a893-be3e-4060-b610-3365757a2b8c-proxy-tls\") pod \"model-chainer-raw-hpa-d5772-75ffc6d89c-k2m2h\" (UID: \"5b48a893-be3e-4060-b610-3365757a2b8c\") " pod="kserve-ci-e2e-test/model-chainer-raw-hpa-d5772-75ffc6d89c-k2m2h"
Sep 17 23:25:21.772799 ip-10-0-142-4 kubenswrapper[2584]: E0917 23:25:21.772795    2584 secret.go:189] Couldn't get secret kserve-ci-e2e-test/model-chainer-raw-hpa-d5772-serving-cert: secret "model-chainer-raw-hpa-d5772-serving-cert" not found
Sep 17 23:25:21.772867 ip-10-0-142-4 kubenswrapper[2584]: E0917 23:25:21.772845    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5b48a893-be3e-4060-b610-3365757a2b8c-proxy-tls podName:5b48a893-be3e-4060-b610-3365757a2b8c nodeName:}" failed. No retries permitted until 2026-09-17 23:25:22.272829633 +0000 UTC m=+905.638489528 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/5b48a893-be3e-4060-b610-3365757a2b8c-proxy-tls") pod "model-chainer-raw-hpa-d5772-75ffc6d89c-k2m2h" (UID: "5b48a893-be3e-4060-b610-3365757a2b8c") : secret "model-chainer-raw-hpa-d5772-serving-cert" not found
Sep 17 23:25:21.773273 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:25:21.773257    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5b48a893-be3e-4060-b610-3365757a2b8c-openshift-service-ca-bundle\") pod \"model-chainer-raw-hpa-d5772-75ffc6d89c-k2m2h\" (UID: \"5b48a893-be3e-4060-b610-3365757a2b8c\") " pod="kserve-ci-e2e-test/model-chainer-raw-hpa-d5772-75ffc6d89c-k2m2h"
Sep 17 23:25:22.040291 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:25:22.040259    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:25:22.277365 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:25:22.277336    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5b48a893-be3e-4060-b610-3365757a2b8c-proxy-tls\") pod \"model-chainer-raw-hpa-d5772-75ffc6d89c-k2m2h\" (UID: \"5b48a893-be3e-4060-b610-3365757a2b8c\") " pod="kserve-ci-e2e-test/model-chainer-raw-hpa-d5772-75ffc6d89c-k2m2h"
Sep 17 23:25:22.279738 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:25:22.279712    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5b48a893-be3e-4060-b610-3365757a2b8c-proxy-tls\") pod \"model-chainer-raw-hpa-d5772-75ffc6d89c-k2m2h\" (UID: \"5b48a893-be3e-4060-b610-3365757a2b8c\") " pod="kserve-ci-e2e-test/model-chainer-raw-hpa-d5772-75ffc6d89c-k2m2h"
Sep 17 23:25:22.477232 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:25:22.477177    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-d5772-75ffc6d89c-k2m2h"
Sep 17 23:25:22.746067 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:25:22.745961    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/model-chainer-raw-hpa-d5772-75ffc6d89c-k2m2h"]
Sep 17 23:25:22.748368 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:25:22.748339    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b48a893_be3e_4060_b610_3365757a2b8c.slice/crio-0bcae653123959bc31da8df622cd80295738fbe1dc350c853bc3bbbd016cfe28 WatchSource:0}: Error finding container 0bcae653123959bc31da8df622cd80295738fbe1dc350c853bc3bbbd016cfe28: Status 404 returned error can't find the container with id 0bcae653123959bc31da8df622cd80295738fbe1dc350c853bc3bbbd016cfe28
Sep 17 23:25:23.675746 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:25:23.675716    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-d5772-75ffc6d89c-k2m2h" event={"ID":"5b48a893-be3e-4060-b610-3365757a2b8c","Type":"ContainerStarted","Data":"ca3a4c4b7d5a3954b491ba5ca56e3a529dbe57a6abf324fd05dfea88a10294a7"}
Sep 17 23:25:23.675913 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:25:23.675765    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-d5772-75ffc6d89c-k2m2h" event={"ID":"5b48a893-be3e-4060-b610-3365757a2b8c","Type":"ContainerStarted","Data":"0bcae653123959bc31da8df622cd80295738fbe1dc350c853bc3bbbd016cfe28"}
Sep 17 23:25:23.675913 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:25:23.675878    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-d5772-75ffc6d89c-k2m2h"
Sep 17 23:25:23.692836 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:25:23.692792    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-d5772-75ffc6d89c-k2m2h" podStartSLOduration=2.692777467 podStartE2EDuration="2.692777467s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:25:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:25:23.691709559 +0000 UTC m=+907.057369477" watchObservedRunningTime="2026-09-17 23:25:23.692777467 +0000 UTC m=+907.058437382"
Sep 17 23:25:29.684875 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:25:29.684849    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-d5772-75ffc6d89c-k2m2h"
Sep 17 23:25:31.636764 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:25:31.636735    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/model-chainer-raw-hpa-d5772-75ffc6d89c-k2m2h"]
Sep 17 23:25:31.637254 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:25:31.636944    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-d5772-75ffc6d89c-k2m2h" podUID="5b48a893-be3e-4060-b610-3365757a2b8c" containerName="model-chainer-raw-hpa-d5772" containerID="cri-o://ca3a4c4b7d5a3954b491ba5ca56e3a529dbe57a6abf324fd05dfea88a10294a7" gracePeriod=30
Sep 17 23:25:31.663883 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:25:31.663859    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:25:31.758937 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:25:31.758903    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-d5772-predictor-54794bf88c-ptxqp"]
Sep 17 23:25:31.759227 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:25:31.759179    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-d5772-predictor-54794bf88c-ptxqp" podUID="4870289b-2f1d-468b-a0ee-c860b460b581" containerName="kserve-container" containerID="cri-o://6c45bf1c18d69351550db4e8a75bfe0f0b17b44a957317117312a0c50bd297e4" gracePeriod=30
Sep 17 23:25:31.759387 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:25:31.759359    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-d5772-predictor-54794bf88c-ptxqp" podUID="4870289b-2f1d-468b-a0ee-c860b460b581" containerName="kube-rbac-proxy" containerID="cri-o://b240b00217329796d85e516a8928c38007e0bd9872c0b04a9326ae079f24d093" gracePeriod=30
Sep 17 23:25:31.769890 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:25:31.769865    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:25:31.829744 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:25:31.829715    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-d5772-predictor-56c57b5777-jrznx"]
Sep 17 23:25:31.829944 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:25:31.829924    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-d5772-predictor-56c57b5777-jrznx" podUID="c490c451-0744-42f3-a5d8-89fe843e7530" containerName="kserve-container" containerID="cri-o://2041d9ebb58c79788e96c90edd9738be9c3576d5bd8111f48602dcb04219fe65" gracePeriod=30
Sep 17 23:25:31.830024 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:25:31.829967    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-d5772-predictor-56c57b5777-jrznx" podUID="c490c451-0744-42f3-a5d8-89fe843e7530" containerName="kube-rbac-proxy" containerID="cri-o://934f451cf6acb5864e1449a0b7864aa9066dcd491c70e9c7b164049f781645c9" gracePeriod=30
Sep 17 23:25:31.873988 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:25:31.873957    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:25:31.956255 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:25:31.956235    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:25:32.718413 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:25:32.718379    2584 generic.go:356] "Generic (PLEG): container finished" podID="4870289b-2f1d-468b-a0ee-c860b460b581" containerID="b240b00217329796d85e516a8928c38007e0bd9872c0b04a9326ae079f24d093" exitCode=2
Sep 17 23:25:32.718784 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:25:32.718449    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-d5772-predictor-54794bf88c-ptxqp" event={"ID":"4870289b-2f1d-468b-a0ee-c860b460b581","Type":"ContainerDied","Data":"b240b00217329796d85e516a8928c38007e0bd9872c0b04a9326ae079f24d093"}
Sep 17 23:25:32.720809 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:25:32.720782    2584 generic.go:356] "Generic (PLEG): container finished" podID="c490c451-0744-42f3-a5d8-89fe843e7530" containerID="934f451cf6acb5864e1449a0b7864aa9066dcd491c70e9c7b164049f781645c9" exitCode=2
Sep 17 23:25:32.720933 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:25:32.720832    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-d5772-predictor-56c57b5777-jrznx" event={"ID":"c490c451-0744-42f3-a5d8-89fe843e7530","Type":"ContainerDied","Data":"934f451cf6acb5864e1449a0b7864aa9066dcd491c70e9c7b164049f781645c9"}
Sep 17 23:25:33.351866 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:25:33.351826    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-d5772-predictor-56c57b5777-jrznx" podUID="c490c451-0744-42f3-a5d8-89fe843e7530" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.25:8643/healthz\": dial tcp 10.132.0.25:8643: connect: connection refused"
Sep 17 23:25:33.356203 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:25:33.356176    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-d5772-predictor-56c57b5777-jrznx" podUID="c490c451-0744-42f3-a5d8-89fe843e7530" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
Sep 17 23:25:34.684075 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:25:34.684035    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-d5772-75ffc6d89c-k2m2h" podUID="5b48a893-be3e-4060-b610-3365757a2b8c" containerName="model-chainer-raw-hpa-d5772" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 23:25:35.270590 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:25:35.270569    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-d5772-predictor-56c57b5777-jrznx"
Sep 17 23:25:35.376562 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:25:35.376488    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c490c451-0744-42f3-a5d8-89fe843e7530-proxy-tls\") pod \"c490c451-0744-42f3-a5d8-89fe843e7530\" (UID: \"c490c451-0744-42f3-a5d8-89fe843e7530\") "
Sep 17 23:25:35.376562 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:25:35.376547    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c490c451-0744-42f3-a5d8-89fe843e7530-cabundle-cert\") pod \"c490c451-0744-42f3-a5d8-89fe843e7530\" (UID: \"c490c451-0744-42f3-a5d8-89fe843e7530\") "
Sep 17 23:25:35.376763 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:25:35.376569    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c490c451-0744-42f3-a5d8-89fe843e7530-kserve-provision-location\") pod \"c490c451-0744-42f3-a5d8-89fe843e7530\" (UID: \"c490c451-0744-42f3-a5d8-89fe843e7530\") "
Sep 17 23:25:35.376763 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:25:35.376588    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sbctp\" (UniqueName: \"kubernetes.io/projected/c490c451-0744-42f3-a5d8-89fe843e7530-kube-api-access-sbctp\") pod \"c490c451-0744-42f3-a5d8-89fe843e7530\" (UID: \"c490c451-0744-42f3-a5d8-89fe843e7530\") "
Sep 17 23:25:35.376763 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:25:35.376665    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c490c451-0744-42f3-a5d8-89fe843e7530-kube-rbac-proxy-sar-config\") pod \"c490c451-0744-42f3-a5d8-89fe843e7530\" (UID: \"c490c451-0744-42f3-a5d8-89fe843e7530\") "
Sep 17 23:25:35.376924 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:25:35.376893    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c490c451-0744-42f3-a5d8-89fe843e7530-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "c490c451-0744-42f3-a5d8-89fe843e7530" (UID: "c490c451-0744-42f3-a5d8-89fe843e7530"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:25:35.377058 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:25:35.376999    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c490c451-0744-42f3-a5d8-89fe843e7530-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "c490c451-0744-42f3-a5d8-89fe843e7530" (UID: "c490c451-0744-42f3-a5d8-89fe843e7530"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:25:35.377351 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:25:35.377073    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c490c451-0744-42f3-a5d8-89fe843e7530-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "c490c451-0744-42f3-a5d8-89fe843e7530" (UID: "c490c451-0744-42f3-a5d8-89fe843e7530"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:25:35.378684 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:25:35.378661    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c490c451-0744-42f3-a5d8-89fe843e7530-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "c490c451-0744-42f3-a5d8-89fe843e7530" (UID: "c490c451-0744-42f3-a5d8-89fe843e7530"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:25:35.378775 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:25:35.378723    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c490c451-0744-42f3-a5d8-89fe843e7530-kube-api-access-sbctp" (OuterVolumeSpecName: "kube-api-access-sbctp") pod "c490c451-0744-42f3-a5d8-89fe843e7530" (UID: "c490c451-0744-42f3-a5d8-89fe843e7530"). InnerVolumeSpecName "kube-api-access-sbctp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:25:35.477239 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:25:35.477210    2584 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c490c451-0744-42f3-a5d8-89fe843e7530-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-4.ec2.internal\" DevicePath \"\""
Sep 17 23:25:35.477239 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:25:35.477236    2584 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c490c451-0744-42f3-a5d8-89fe843e7530-proxy-tls\") on node \"ip-10-0-142-4.ec2.internal\" DevicePath \"\""
Sep 17 23:25:35.477239 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:25:35.477245    2584 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c490c451-0744-42f3-a5d8-89fe843e7530-cabundle-cert\") on node \"ip-10-0-142-4.ec2.internal\" DevicePath \"\""
Sep 17 23:25:35.477459 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:25:35.477254    2584 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c490c451-0744-42f3-a5d8-89fe843e7530-kserve-provision-location\") on node \"ip-10-0-142-4.ec2.internal\" DevicePath \"\""
Sep 17 23:25:35.477459 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:25:35.477262    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sbctp\" (UniqueName: \"kubernetes.io/projected/c490c451-0744-42f3-a5d8-89fe843e7530-kube-api-access-sbctp\") on node \"ip-10-0-142-4.ec2.internal\" DevicePath \"\""
Sep 17 23:25:35.736924 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:25:35.736899    2584 generic.go:356] "Generic (PLEG): container finished" podID="c490c451-0744-42f3-a5d8-89fe843e7530" containerID="2041d9ebb58c79788e96c90edd9738be9c3576d5bd8111f48602dcb04219fe65" exitCode=0
Sep 17 23:25:35.737233 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:25:35.736966    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-d5772-predictor-56c57b5777-jrznx"
Sep 17 23:25:35.737233 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:25:35.736978    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-d5772-predictor-56c57b5777-jrznx" event={"ID":"c490c451-0744-42f3-a5d8-89fe843e7530","Type":"ContainerDied","Data":"2041d9ebb58c79788e96c90edd9738be9c3576d5bd8111f48602dcb04219fe65"}
Sep 17 23:25:35.737233 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:25:35.737073    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-d5772-predictor-56c57b5777-jrznx" event={"ID":"c490c451-0744-42f3-a5d8-89fe843e7530","Type":"ContainerDied","Data":"83004f678aae3e54654e1a0ebd6310cdbcfbf296fe7aacbf857541785b733d38"}
Sep 17 23:25:35.737233 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:25:35.737114    2584 scope.go:117] "RemoveContainer" containerID="934f451cf6acb5864e1449a0b7864aa9066dcd491c70e9c7b164049f781645c9"
Sep 17 23:25:35.781647 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:25:35.781626    2584 scope.go:117] "RemoveContainer" containerID="2041d9ebb58c79788e96c90edd9738be9c3576d5bd8111f48602dcb04219fe65"
Sep 17 23:25:35.789264 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:25:35.789244    2584 scope.go:117] "RemoveContainer" containerID="86fd5096362a6af8899b59a3fc3ee00f2d23ba3dd648f8b80f2878c53b24e01e"
Sep 17 23:25:35.797051 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:25:35.797024    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-d5772-predictor-56c57b5777-jrznx"]
Sep 17 23:25:35.797605 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:25:35.797582    2584 scope.go:117] "RemoveContainer" containerID="934f451cf6acb5864e1449a0b7864aa9066dcd491c70e9c7b164049f781645c9"
Sep 17 23:25:35.797898 ip-10-0-142-4 kubenswrapper[2584]: E0917 23:25:35.797875    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"934f451cf6acb5864e1449a0b7864aa9066dcd491c70e9c7b164049f781645c9\": container with ID starting with 934f451cf6acb5864e1449a0b7864aa9066dcd491c70e9c7b164049f781645c9 not found: ID does not exist" containerID="934f451cf6acb5864e1449a0b7864aa9066dcd491c70e9c7b164049f781645c9"
Sep 17 23:25:35.797976 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:25:35.797907    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"934f451cf6acb5864e1449a0b7864aa9066dcd491c70e9c7b164049f781645c9"} err="failed to get container status \"934f451cf6acb5864e1449a0b7864aa9066dcd491c70e9c7b164049f781645c9\": rpc error: code = NotFound desc = could not find container \"934f451cf6acb5864e1449a0b7864aa9066dcd491c70e9c7b164049f781645c9\": container with ID starting with 934f451cf6acb5864e1449a0b7864aa9066dcd491c70e9c7b164049f781645c9 not found: ID does not exist"
Sep 17 23:25:35.797976 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:25:35.797925    2584 scope.go:117] "RemoveContainer" containerID="2041d9ebb58c79788e96c90edd9738be9c3576d5bd8111f48602dcb04219fe65"
Sep 17 23:25:35.798273 ip-10-0-142-4 kubenswrapper[2584]: E0917 23:25:35.798255    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2041d9ebb58c79788e96c90edd9738be9c3576d5bd8111f48602dcb04219fe65\": container with ID starting with 2041d9ebb58c79788e96c90edd9738be9c3576d5bd8111f48602dcb04219fe65 not found: ID does not exist" containerID="2041d9ebb58c79788e96c90edd9738be9c3576d5bd8111f48602dcb04219fe65"
Sep 17 23:25:35.798321 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:25:35.798280    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2041d9ebb58c79788e96c90edd9738be9c3576d5bd8111f48602dcb04219fe65"} err="failed to get container status \"2041d9ebb58c79788e96c90edd9738be9c3576d5bd8111f48602dcb04219fe65\": rpc error: code = NotFound desc = could not find container \"2041d9ebb58c79788e96c90edd9738be9c3576d5bd8111f48602dcb04219fe65\": container with ID starting with 2041d9ebb58c79788e96c90edd9738be9c3576d5bd8111f48602dcb04219fe65 not found: ID does not exist"
Sep 17 23:25:35.798321 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:25:35.798295    2584 scope.go:117] "RemoveContainer" containerID="86fd5096362a6af8899b59a3fc3ee00f2d23ba3dd648f8b80f2878c53b24e01e"
Sep 17 23:25:35.798743 ip-10-0-142-4 kubenswrapper[2584]: E0917 23:25:35.798683    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"86fd5096362a6af8899b59a3fc3ee00f2d23ba3dd648f8b80f2878c53b24e01e\": container with ID starting with 86fd5096362a6af8899b59a3fc3ee00f2d23ba3dd648f8b80f2878c53b24e01e not found: ID does not exist" containerID="86fd5096362a6af8899b59a3fc3ee00f2d23ba3dd648f8b80f2878c53b24e01e"
Sep 17 23:25:35.798743 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:25:35.798718    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86fd5096362a6af8899b59a3fc3ee00f2d23ba3dd648f8b80f2878c53b24e01e"} err="failed to get container status \"86fd5096362a6af8899b59a3fc3ee00f2d23ba3dd648f8b80f2878c53b24e01e\": rpc error: code = NotFound desc = could not find container \"86fd5096362a6af8899b59a3fc3ee00f2d23ba3dd648f8b80f2878c53b24e01e\": container with ID starting with 86fd5096362a6af8899b59a3fc3ee00f2d23ba3dd648f8b80f2878c53b24e01e not found: ID does not exist"
Sep 17 23:25:35.800933 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:25:35.800912    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-d5772-predictor-56c57b5777-jrznx"]
Sep 17 23:25:35.887867 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:25:35.887846    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-d5772-predictor-54794bf88c-ptxqp"
Sep 17 23:25:35.981459 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:25:35.981373    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4870289b-2f1d-468b-a0ee-c860b460b581-kserve-provision-location\") pod \"4870289b-2f1d-468b-a0ee-c860b460b581\" (UID: \"4870289b-2f1d-468b-a0ee-c860b460b581\") "
Sep 17 23:25:35.981459 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:25:35.981412    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4870289b-2f1d-468b-a0ee-c860b460b581-cabundle-cert\") pod \"4870289b-2f1d-468b-a0ee-c860b460b581\" (UID: \"4870289b-2f1d-468b-a0ee-c860b460b581\") "
Sep 17 23:25:35.981642 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:25:35.981475    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cf2jv\" (UniqueName: \"kubernetes.io/projected/4870289b-2f1d-468b-a0ee-c860b460b581-kube-api-access-cf2jv\") pod \"4870289b-2f1d-468b-a0ee-c860b460b581\" (UID: \"4870289b-2f1d-468b-a0ee-c860b460b581\") "
Sep 17 23:25:35.981642 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:25:35.981499    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4870289b-2f1d-468b-a0ee-c860b460b581-kube-rbac-proxy-sar-config\") pod \"4870289b-2f1d-468b-a0ee-c860b460b581\" (UID: \"4870289b-2f1d-468b-a0ee-c860b460b581\") "
Sep 17 23:25:35.981642 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:25:35.981548    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4870289b-2f1d-468b-a0ee-c860b460b581-proxy-tls\") pod \"4870289b-2f1d-468b-a0ee-c860b460b581\" (UID: \"4870289b-2f1d-468b-a0ee-c860b460b581\") "
Sep 17 23:25:35.981762 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:25:35.981740    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4870289b-2f1d-468b-a0ee-c860b460b581-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "4870289b-2f1d-468b-a0ee-c860b460b581" (UID: "4870289b-2f1d-468b-a0ee-c860b460b581"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:25:35.981900 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:25:35.981873    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4870289b-2f1d-468b-a0ee-c860b460b581-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "4870289b-2f1d-468b-a0ee-c860b460b581" (UID: "4870289b-2f1d-468b-a0ee-c860b460b581"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:25:35.981961 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:25:35.981931    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4870289b-2f1d-468b-a0ee-c860b460b581-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "4870289b-2f1d-468b-a0ee-c860b460b581" (UID: "4870289b-2f1d-468b-a0ee-c860b460b581"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:25:35.983697 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:25:35.983678    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4870289b-2f1d-468b-a0ee-c860b460b581-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "4870289b-2f1d-468b-a0ee-c860b460b581" (UID: "4870289b-2f1d-468b-a0ee-c860b460b581"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:25:35.983769 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:25:35.983712    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4870289b-2f1d-468b-a0ee-c860b460b581-kube-api-access-cf2jv" (OuterVolumeSpecName: "kube-api-access-cf2jv") pod "4870289b-2f1d-468b-a0ee-c860b460b581" (UID: "4870289b-2f1d-468b-a0ee-c860b460b581"). InnerVolumeSpecName "kube-api-access-cf2jv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:25:36.082793 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:25:36.082758    2584 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4870289b-2f1d-468b-a0ee-c860b460b581-kserve-provision-location\") on node \"ip-10-0-142-4.ec2.internal\" DevicePath \"\""
Sep 17 23:25:36.082793 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:25:36.082786    2584 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4870289b-2f1d-468b-a0ee-c860b460b581-cabundle-cert\") on node \"ip-10-0-142-4.ec2.internal\" DevicePath \"\""
Sep 17 23:25:36.082793 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:25:36.082796    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cf2jv\" (UniqueName: \"kubernetes.io/projected/4870289b-2f1d-468b-a0ee-c860b460b581-kube-api-access-cf2jv\") on node \"ip-10-0-142-4.ec2.internal\" DevicePath \"\""
Sep 17 23:25:36.083057 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:25:36.082817    2584 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4870289b-2f1d-468b-a0ee-c860b460b581-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-4.ec2.internal\" DevicePath \"\""
Sep 17 23:25:36.083057 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:25:36.082826    2584 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4870289b-2f1d-468b-a0ee-c860b460b581-proxy-tls\") on node \"ip-10-0-142-4.ec2.internal\" DevicePath \"\""
Sep 17 23:25:36.743172 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:25:36.743140    2584 generic.go:356] "Generic (PLEG): container finished" podID="4870289b-2f1d-468b-a0ee-c860b460b581" containerID="6c45bf1c18d69351550db4e8a75bfe0f0b17b44a957317117312a0c50bd297e4" exitCode=0
Sep 17 23:25:36.743617 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:25:36.743191    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-d5772-predictor-54794bf88c-ptxqp"
Sep 17 23:25:36.743617 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:25:36.743228    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-d5772-predictor-54794bf88c-ptxqp" event={"ID":"4870289b-2f1d-468b-a0ee-c860b460b581","Type":"ContainerDied","Data":"6c45bf1c18d69351550db4e8a75bfe0f0b17b44a957317117312a0c50bd297e4"}
Sep 17 23:25:36.743617 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:25:36.743305    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-d5772-predictor-54794bf88c-ptxqp" event={"ID":"4870289b-2f1d-468b-a0ee-c860b460b581","Type":"ContainerDied","Data":"e4ad8451d5d0a84cbaf3e0d1c9b40eb4abb743a80f4c9d4f387f1b9349bdb621"}
Sep 17 23:25:36.743617 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:25:36.743326    2584 scope.go:117] "RemoveContainer" containerID="b240b00217329796d85e516a8928c38007e0bd9872c0b04a9326ae079f24d093"
Sep 17 23:25:36.751520 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:25:36.751506    2584 scope.go:117] "RemoveContainer" containerID="6c45bf1c18d69351550db4e8a75bfe0f0b17b44a957317117312a0c50bd297e4"
Sep 17 23:25:36.759166 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:25:36.759146    2584 scope.go:117] "RemoveContainer" containerID="76d931b06d543340959f1bfcf7bb893d73864e78e7e8b12392afa7e633723095"
Sep 17 23:25:36.763816 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:25:36.763787    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c490c451-0744-42f3-a5d8-89fe843e7530" path="/var/lib/kubelet/pods/c490c451-0744-42f3-a5d8-89fe843e7530/volumes"
Sep 17 23:25:36.764467 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:25:36.764442    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-d5772-predictor-54794bf88c-ptxqp"]
Sep 17 23:25:36.767124 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:25:36.767103    2584 scope.go:117] "RemoveContainer" containerID="b240b00217329796d85e516a8928c38007e0bd9872c0b04a9326ae079f24d093"
Sep 17 23:25:36.767404 ip-10-0-142-4 kubenswrapper[2584]: E0917 23:25:36.767385    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b240b00217329796d85e516a8928c38007e0bd9872c0b04a9326ae079f24d093\": container with ID starting with b240b00217329796d85e516a8928c38007e0bd9872c0b04a9326ae079f24d093 not found: ID does not exist" containerID="b240b00217329796d85e516a8928c38007e0bd9872c0b04a9326ae079f24d093"
Sep 17 23:25:36.767463 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:25:36.767416    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b240b00217329796d85e516a8928c38007e0bd9872c0b04a9326ae079f24d093"} err="failed to get container status \"b240b00217329796d85e516a8928c38007e0bd9872c0b04a9326ae079f24d093\": rpc error: code = NotFound desc = could not find container \"b240b00217329796d85e516a8928c38007e0bd9872c0b04a9326ae079f24d093\": container with ID starting with b240b00217329796d85e516a8928c38007e0bd9872c0b04a9326ae079f24d093 not found: ID does not exist"
Sep 17 23:25:36.767463 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:25:36.767434    2584 scope.go:117] "RemoveContainer" containerID="6c45bf1c18d69351550db4e8a75bfe0f0b17b44a957317117312a0c50bd297e4"
Sep 17 23:25:36.767716 ip-10-0-142-4 kubenswrapper[2584]: E0917 23:25:36.767702    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c45bf1c18d69351550db4e8a75bfe0f0b17b44a957317117312a0c50bd297e4\": container with ID starting with 6c45bf1c18d69351550db4e8a75bfe0f0b17b44a957317117312a0c50bd297e4 not found: ID does not exist" containerID="6c45bf1c18d69351550db4e8a75bfe0f0b17b44a957317117312a0c50bd297e4"
Sep 17 23:25:36.767763 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:25:36.767719    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c45bf1c18d69351550db4e8a75bfe0f0b17b44a957317117312a0c50bd297e4"} err="failed to get container status \"6c45bf1c18d69351550db4e8a75bfe0f0b17b44a957317117312a0c50bd297e4\": rpc error: code = NotFound desc = could not find container \"6c45bf1c18d69351550db4e8a75bfe0f0b17b44a957317117312a0c50bd297e4\": container with ID starting with 6c45bf1c18d69351550db4e8a75bfe0f0b17b44a957317117312a0c50bd297e4 not found: ID does not exist"
Sep 17 23:25:36.767763 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:25:36.767731    2584 scope.go:117] "RemoveContainer" containerID="76d931b06d543340959f1bfcf7bb893d73864e78e7e8b12392afa7e633723095"
Sep 17 23:25:36.768204 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:25:36.768186    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-d5772-predictor-54794bf88c-ptxqp"]
Sep 17 23:25:36.768306 ip-10-0-142-4 kubenswrapper[2584]: E0917 23:25:36.768282    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"76d931b06d543340959f1bfcf7bb893d73864e78e7e8b12392afa7e633723095\": container with ID starting with 76d931b06d543340959f1bfcf7bb893d73864e78e7e8b12392afa7e633723095 not found: ID does not exist" containerID="76d931b06d543340959f1bfcf7bb893d73864e78e7e8b12392afa7e633723095"
Sep 17 23:25:36.768359 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:25:36.768317    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"76d931b06d543340959f1bfcf7bb893d73864e78e7e8b12392afa7e633723095"} err="failed to get container status \"76d931b06d543340959f1bfcf7bb893d73864e78e7e8b12392afa7e633723095\": rpc error: code = NotFound desc = could not find container \"76d931b06d543340959f1bfcf7bb893d73864e78e7e8b12392afa7e633723095\": container with ID starting with 76d931b06d543340959f1bfcf7bb893d73864e78e7e8b12392afa7e633723095 not found: ID does not exist"
Sep 17 23:25:38.762265 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:25:38.762234    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4870289b-2f1d-468b-a0ee-c860b460b581" path="/var/lib/kubelet/pods/4870289b-2f1d-468b-a0ee-c860b460b581/volumes"
Sep 17 23:25:39.684029 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:25:39.683973    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-d5772-75ffc6d89c-k2m2h" podUID="5b48a893-be3e-4060-b610-3365757a2b8c" containerName="model-chainer-raw-hpa-d5772" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 23:25:44.684366 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:25:44.684328    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-d5772-75ffc6d89c-k2m2h" podUID="5b48a893-be3e-4060-b610-3365757a2b8c" containerName="model-chainer-raw-hpa-d5772" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 23:25:44.684715 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:25:44.684425    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-d5772-75ffc6d89c-k2m2h"
Sep 17 23:25:49.684588 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:25:49.684545    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-d5772-75ffc6d89c-k2m2h" podUID="5b48a893-be3e-4060-b610-3365757a2b8c" containerName="model-chainer-raw-hpa-d5772" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 23:25:51.884838 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:25:51.884808    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:25:51.928248 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:25:51.928219    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-0a0da-predictor-64d6ff776d-ccvv9"]
Sep 17 23:25:51.928530 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:25:51.928518    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4870289b-2f1d-468b-a0ee-c860b460b581" containerName="kube-rbac-proxy"
Sep 17 23:25:51.928577 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:25:51.928532    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="4870289b-2f1d-468b-a0ee-c860b460b581" containerName="kube-rbac-proxy"
Sep 17 23:25:51.928577 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:25:51.928547    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c490c451-0744-42f3-a5d8-89fe843e7530" containerName="kserve-container"
Sep 17 23:25:51.928577 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:25:51.928553    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="c490c451-0744-42f3-a5d8-89fe843e7530" containerName="kserve-container"
Sep 17 23:25:51.928577 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:25:51.928560    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4870289b-2f1d-468b-a0ee-c860b460b581" containerName="storage-initializer"
Sep 17 23:25:51.928577 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:25:51.928565    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="4870289b-2f1d-468b-a0ee-c860b460b581" containerName="storage-initializer"
Sep 17 23:25:51.928577 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:25:51.928572    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4870289b-2f1d-468b-a0ee-c860b460b581" containerName="kserve-container"
Sep 17 23:25:51.928577 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:25:51.928577    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="4870289b-2f1d-468b-a0ee-c860b460b581" containerName="kserve-container"
Sep 17 23:25:51.928791 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:25:51.928589    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c490c451-0744-42f3-a5d8-89fe843e7530" containerName="kube-rbac-proxy"
Sep 17 23:25:51.928791 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:25:51.928594    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="c490c451-0744-42f3-a5d8-89fe843e7530" containerName="kube-rbac-proxy"
Sep 17 23:25:51.928791 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:25:51.928602    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c490c451-0744-42f3-a5d8-89fe843e7530" containerName="storage-initializer"
Sep 17 23:25:51.928791 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:25:51.928607    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="c490c451-0744-42f3-a5d8-89fe843e7530" containerName="storage-initializer"
Sep 17 23:25:51.928791 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:25:51.928650    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="c490c451-0744-42f3-a5d8-89fe843e7530" containerName="kube-rbac-proxy"
Sep 17 23:25:51.928791 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:25:51.928657    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="4870289b-2f1d-468b-a0ee-c860b460b581" containerName="kube-rbac-proxy"
Sep 17 23:25:51.928791 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:25:51.928665    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="c490c451-0744-42f3-a5d8-89fe843e7530" containerName="kserve-container"
Sep 17 23:25:51.928791 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:25:51.928672    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="4870289b-2f1d-468b-a0ee-c860b460b581" containerName="kserve-container"
Sep 17 23:25:51.933567 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:25:51.933552    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-0a0da-predictor-64d6ff776d-ccvv9"
Sep 17 23:25:51.936216 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:25:51.936193    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-raw-0a0da-predictor-serving-cert\""
Sep 17 23:25:51.936325 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:25:51.936222    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 23:25:51.937423 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:25:51.937408    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 23:25:51.937657 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:25:51.937642    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 23:25:51.937716 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:25:51.937703    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-raw-0a0da-kube-rbac-proxy-sar-config\""
Sep 17 23:25:51.945329 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:25:51.945307    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-0a0da-predictor-64d6ff776d-ccvv9"]
Sep 17 23:25:51.997286 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:25:51.997251    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/42bb7c6e-5c8f-466d-b6db-d6c2e6b65e68-proxy-tls\") pod \"isvc-logger-raw-0a0da-predictor-64d6ff776d-ccvv9\" (UID: \"42bb7c6e-5c8f-466d-b6db-d6c2e6b65e68\") " pod="kserve-ci-e2e-test/isvc-logger-raw-0a0da-predictor-64d6ff776d-ccvv9"
Sep 17 23:25:51.997286 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:25:51.997296    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/42bb7c6e-5c8f-466d-b6db-d6c2e6b65e68-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-0a0da-predictor-64d6ff776d-ccvv9\" (UID: \"42bb7c6e-5c8f-466d-b6db-d6c2e6b65e68\") " pod="kserve-ci-e2e-test/isvc-logger-raw-0a0da-predictor-64d6ff776d-ccvv9"
Sep 17 23:25:51.997517 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:25:51.997312    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gqffc\" (UniqueName: \"kubernetes.io/projected/42bb7c6e-5c8f-466d-b6db-d6c2e6b65e68-kube-api-access-gqffc\") pod \"isvc-logger-raw-0a0da-predictor-64d6ff776d-ccvv9\" (UID: \"42bb7c6e-5c8f-466d-b6db-d6c2e6b65e68\") " pod="kserve-ci-e2e-test/isvc-logger-raw-0a0da-predictor-64d6ff776d-ccvv9"
Sep 17 23:25:51.997517 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:25:51.997332    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/42bb7c6e-5c8f-466d-b6db-d6c2e6b65e68-cabundle-cert\") pod \"isvc-logger-raw-0a0da-predictor-64d6ff776d-ccvv9\" (UID: \"42bb7c6e-5c8f-466d-b6db-d6c2e6b65e68\") " pod="kserve-ci-e2e-test/isvc-logger-raw-0a0da-predictor-64d6ff776d-ccvv9"
Sep 17 23:25:51.997517 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:25:51.997351    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/42bb7c6e-5c8f-466d-b6db-d6c2e6b65e68-kserve-provision-location\") pod \"isvc-logger-raw-0a0da-predictor-64d6ff776d-ccvv9\" (UID: \"42bb7c6e-5c8f-466d-b6db-d6c2e6b65e68\") " pod="kserve-ci-e2e-test/isvc-logger-raw-0a0da-predictor-64d6ff776d-ccvv9"
Sep 17 23:25:52.062555 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:25:52.062522    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:25:52.098722 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:25:52.098688    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/42bb7c6e-5c8f-466d-b6db-d6c2e6b65e68-cabundle-cert\") pod \"isvc-logger-raw-0a0da-predictor-64d6ff776d-ccvv9\" (UID: \"42bb7c6e-5c8f-466d-b6db-d6c2e6b65e68\") " pod="kserve-ci-e2e-test/isvc-logger-raw-0a0da-predictor-64d6ff776d-ccvv9"
Sep 17 23:25:52.098722 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:25:52.098725    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/42bb7c6e-5c8f-466d-b6db-d6c2e6b65e68-kserve-provision-location\") pod \"isvc-logger-raw-0a0da-predictor-64d6ff776d-ccvv9\" (UID: \"42bb7c6e-5c8f-466d-b6db-d6c2e6b65e68\") " pod="kserve-ci-e2e-test/isvc-logger-raw-0a0da-predictor-64d6ff776d-ccvv9"
Sep 17 23:25:52.098961 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:25:52.098791    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/42bb7c6e-5c8f-466d-b6db-d6c2e6b65e68-proxy-tls\") pod \"isvc-logger-raw-0a0da-predictor-64d6ff776d-ccvv9\" (UID: \"42bb7c6e-5c8f-466d-b6db-d6c2e6b65e68\") " pod="kserve-ci-e2e-test/isvc-logger-raw-0a0da-predictor-64d6ff776d-ccvv9"
Sep 17 23:25:52.098961 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:25:52.098816    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/42bb7c6e-5c8f-466d-b6db-d6c2e6b65e68-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-0a0da-predictor-64d6ff776d-ccvv9\" (UID: \"42bb7c6e-5c8f-466d-b6db-d6c2e6b65e68\") " pod="kserve-ci-e2e-test/isvc-logger-raw-0a0da-predictor-64d6ff776d-ccvv9"
Sep 17 23:25:52.098961 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:25:52.098831    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gqffc\" (UniqueName: \"kubernetes.io/projected/42bb7c6e-5c8f-466d-b6db-d6c2e6b65e68-kube-api-access-gqffc\") pod \"isvc-logger-raw-0a0da-predictor-64d6ff776d-ccvv9\" (UID: \"42bb7c6e-5c8f-466d-b6db-d6c2e6b65e68\") " pod="kserve-ci-e2e-test/isvc-logger-raw-0a0da-predictor-64d6ff776d-ccvv9"
Sep 17 23:25:52.099261 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:25:52.099238    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/42bb7c6e-5c8f-466d-b6db-d6c2e6b65e68-kserve-provision-location\") pod \"isvc-logger-raw-0a0da-predictor-64d6ff776d-ccvv9\" (UID: \"42bb7c6e-5c8f-466d-b6db-d6c2e6b65e68\") " pod="kserve-ci-e2e-test/isvc-logger-raw-0a0da-predictor-64d6ff776d-ccvv9"
Sep 17 23:25:52.099544 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:25:52.099524    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/42bb7c6e-5c8f-466d-b6db-d6c2e6b65e68-cabundle-cert\") pod \"isvc-logger-raw-0a0da-predictor-64d6ff776d-ccvv9\" (UID: \"42bb7c6e-5c8f-466d-b6db-d6c2e6b65e68\") " pod="kserve-ci-e2e-test/isvc-logger-raw-0a0da-predictor-64d6ff776d-ccvv9"
Sep 17 23:25:52.099585 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:25:52.099524    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/42bb7c6e-5c8f-466d-b6db-d6c2e6b65e68-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-0a0da-predictor-64d6ff776d-ccvv9\" (UID: \"42bb7c6e-5c8f-466d-b6db-d6c2e6b65e68\") " pod="kserve-ci-e2e-test/isvc-logger-raw-0a0da-predictor-64d6ff776d-ccvv9"
Sep 17 23:25:52.101456 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:25:52.101434    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/42bb7c6e-5c8f-466d-b6db-d6c2e6b65e68-proxy-tls\") pod \"isvc-logger-raw-0a0da-predictor-64d6ff776d-ccvv9\" (UID: \"42bb7c6e-5c8f-466d-b6db-d6c2e6b65e68\") " pod="kserve-ci-e2e-test/isvc-logger-raw-0a0da-predictor-64d6ff776d-ccvv9"
Sep 17 23:25:52.108165 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:25:52.108141    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gqffc\" (UniqueName: \"kubernetes.io/projected/42bb7c6e-5c8f-466d-b6db-d6c2e6b65e68-kube-api-access-gqffc\") pod \"isvc-logger-raw-0a0da-predictor-64d6ff776d-ccvv9\" (UID: \"42bb7c6e-5c8f-466d-b6db-d6c2e6b65e68\") " pod="kserve-ci-e2e-test/isvc-logger-raw-0a0da-predictor-64d6ff776d-ccvv9"
Sep 17 23:25:52.243747 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:25:52.243667    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-0a0da-predictor-64d6ff776d-ccvv9"
Sep 17 23:25:52.518191 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:25:52.518132    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-0a0da-predictor-64d6ff776d-ccvv9"]
Sep 17 23:25:52.520405 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:25:52.520376    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42bb7c6e_5c8f_466d_b6db_d6c2e6b65e68.slice/crio-1b22df02beee00a2fcf37b222acf042eaa789f5607eb579e9c2efdd034dd374d WatchSource:0}: Error finding container 1b22df02beee00a2fcf37b222acf042eaa789f5607eb579e9c2efdd034dd374d: Status 404 returned error can't find the container with id 1b22df02beee00a2fcf37b222acf042eaa789f5607eb579e9c2efdd034dd374d
Sep 17 23:25:52.806251 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:25:52.806175    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-0a0da-predictor-64d6ff776d-ccvv9" event={"ID":"42bb7c6e-5c8f-466d-b6db-d6c2e6b65e68","Type":"ContainerStarted","Data":"bfe5a864fcdc035838f1c9ce419b6e9ad400210b07715776795bfeaf833a3342"}
Sep 17 23:25:52.806251 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:25:52.806231    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-0a0da-predictor-64d6ff776d-ccvv9" event={"ID":"42bb7c6e-5c8f-466d-b6db-d6c2e6b65e68","Type":"ContainerStarted","Data":"1b22df02beee00a2fcf37b222acf042eaa789f5607eb579e9c2efdd034dd374d"}
Sep 17 23:25:54.684286 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:25:54.684250    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-d5772-75ffc6d89c-k2m2h" podUID="5b48a893-be3e-4060-b610-3365757a2b8c" containerName="model-chainer-raw-hpa-d5772" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 23:25:56.823040 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:25:56.822996    2584 generic.go:356] "Generic (PLEG): container finished" podID="42bb7c6e-5c8f-466d-b6db-d6c2e6b65e68" containerID="bfe5a864fcdc035838f1c9ce419b6e9ad400210b07715776795bfeaf833a3342" exitCode=0
Sep 17 23:25:56.823417 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:25:56.823066    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-0a0da-predictor-64d6ff776d-ccvv9" event={"ID":"42bb7c6e-5c8f-466d-b6db-d6c2e6b65e68","Type":"ContainerDied","Data":"bfe5a864fcdc035838f1c9ce419b6e9ad400210b07715776795bfeaf833a3342"}
Sep 17 23:25:57.831059 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:25:57.831023    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-0a0da-predictor-64d6ff776d-ccvv9" event={"ID":"42bb7c6e-5c8f-466d-b6db-d6c2e6b65e68","Type":"ContainerStarted","Data":"46921f616f8e8fc0e4a5802ca18a2f9b8d9516260fd3d1ca5b40a79bc620a9e3"}
Sep 17 23:25:57.831394 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:25:57.831096    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-0a0da-predictor-64d6ff776d-ccvv9" event={"ID":"42bb7c6e-5c8f-466d-b6db-d6c2e6b65e68","Type":"ContainerStarted","Data":"4beb4c7ba12dbb8275a3b02db7c2564c61fc8eaee5b48f582a7af6fc8df2de10"}
Sep 17 23:25:57.831394 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:25:57.831123    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-0a0da-predictor-64d6ff776d-ccvv9" event={"ID":"42bb7c6e-5c8f-466d-b6db-d6c2e6b65e68","Type":"ContainerStarted","Data":"5fabeefb40484c77f2cf6c637dd0c949116d093312844bfb10481a169b9a997a"}
Sep 17 23:25:57.831489 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:25:57.831406    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-0a0da-predictor-64d6ff776d-ccvv9"
Sep 17 23:25:57.831577 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:25:57.831559    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-0a0da-predictor-64d6ff776d-ccvv9"
Sep 17 23:25:57.832501 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:25:57.832473    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-0a0da-predictor-64d6ff776d-ccvv9" podUID="42bb7c6e-5c8f-466d-b6db-d6c2e6b65e68" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Sep 17 23:25:57.855595 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:25:57.855554    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-logger-raw-0a0da-predictor-64d6ff776d-ccvv9" podStartSLOduration=3.184584514 podStartE2EDuration="6.855541018s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.670956504s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:25:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:25:57.854812003 +0000 UTC m=+941.220471930" watchObservedRunningTime="2026-09-17 23:25:57.855541018 +0000 UTC m=+941.221200931"
Sep 17 23:25:58.835288 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:25:58.835257    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-0a0da-predictor-64d6ff776d-ccvv9"
Sep 17 23:25:58.835640 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:25:58.835256    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-0a0da-predictor-64d6ff776d-ccvv9" podUID="42bb7c6e-5c8f-466d-b6db-d6c2e6b65e68" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Sep 17 23:25:58.836309 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:25:58.836282    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-0a0da-predictor-64d6ff776d-ccvv9" podUID="42bb7c6e-5c8f-466d-b6db-d6c2e6b65e68" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 23:25:59.683468 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:25:59.683424    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-d5772-75ffc6d89c-k2m2h" podUID="5b48a893-be3e-4060-b610-3365757a2b8c" containerName="model-chainer-raw-hpa-d5772" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 23:25:59.840070 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:25:59.840001    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-0a0da-predictor-64d6ff776d-ccvv9" podUID="42bb7c6e-5c8f-466d-b6db-d6c2e6b65e68" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Sep 17 23:25:59.840507 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:25:59.840482    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-0a0da-predictor-64d6ff776d-ccvv9" podUID="42bb7c6e-5c8f-466d-b6db-d6c2e6b65e68" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 23:26:01.820066 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:26:01.820040    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-d5772-75ffc6d89c-k2m2h"
Sep 17 23:26:01.848666 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:26:01.848639    2584 generic.go:356] "Generic (PLEG): container finished" podID="5b48a893-be3e-4060-b610-3365757a2b8c" containerID="ca3a4c4b7d5a3954b491ba5ca56e3a529dbe57a6abf324fd05dfea88a10294a7" exitCode=0
Sep 17 23:26:01.848795 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:26:01.848694    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-d5772-75ffc6d89c-k2m2h"
Sep 17 23:26:01.848795 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:26:01.848729    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-d5772-75ffc6d89c-k2m2h" event={"ID":"5b48a893-be3e-4060-b610-3365757a2b8c","Type":"ContainerDied","Data":"ca3a4c4b7d5a3954b491ba5ca56e3a529dbe57a6abf324fd05dfea88a10294a7"}
Sep 17 23:26:01.848795 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:26:01.848792    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-d5772-75ffc6d89c-k2m2h" event={"ID":"5b48a893-be3e-4060-b610-3365757a2b8c","Type":"ContainerDied","Data":"0bcae653123959bc31da8df622cd80295738fbe1dc350c853bc3bbbd016cfe28"}
Sep 17 23:26:01.848955 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:26:01.848822    2584 scope.go:117] "RemoveContainer" containerID="ca3a4c4b7d5a3954b491ba5ca56e3a529dbe57a6abf324fd05dfea88a10294a7"
Sep 17 23:26:01.857046 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:26:01.857031    2584 scope.go:117] "RemoveContainer" containerID="ca3a4c4b7d5a3954b491ba5ca56e3a529dbe57a6abf324fd05dfea88a10294a7"
Sep 17 23:26:01.857306 ip-10-0-142-4 kubenswrapper[2584]: E0917 23:26:01.857289    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca3a4c4b7d5a3954b491ba5ca56e3a529dbe57a6abf324fd05dfea88a10294a7\": container with ID starting with ca3a4c4b7d5a3954b491ba5ca56e3a529dbe57a6abf324fd05dfea88a10294a7 not found: ID does not exist" containerID="ca3a4c4b7d5a3954b491ba5ca56e3a529dbe57a6abf324fd05dfea88a10294a7"
Sep 17 23:26:01.857361 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:26:01.857313    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca3a4c4b7d5a3954b491ba5ca56e3a529dbe57a6abf324fd05dfea88a10294a7"} err="failed to get container status \"ca3a4c4b7d5a3954b491ba5ca56e3a529dbe57a6abf324fd05dfea88a10294a7\": rpc error: code = NotFound desc = could not find container \"ca3a4c4b7d5a3954b491ba5ca56e3a529dbe57a6abf324fd05dfea88a10294a7\": container with ID starting with ca3a4c4b7d5a3954b491ba5ca56e3a529dbe57a6abf324fd05dfea88a10294a7 not found: ID does not exist"
Sep 17 23:26:01.875589 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:26:01.875569    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5b48a893-be3e-4060-b610-3365757a2b8c-openshift-service-ca-bundle\") pod \"5b48a893-be3e-4060-b610-3365757a2b8c\" (UID: \"5b48a893-be3e-4060-b610-3365757a2b8c\") "
Sep 17 23:26:01.875661 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:26:01.875606    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5b48a893-be3e-4060-b610-3365757a2b8c-proxy-tls\") pod \"5b48a893-be3e-4060-b610-3365757a2b8c\" (UID: \"5b48a893-be3e-4060-b610-3365757a2b8c\") "
Sep 17 23:26:01.875866 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:26:01.875844    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b48a893-be3e-4060-b610-3365757a2b8c-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "5b48a893-be3e-4060-b610-3365757a2b8c" (UID: "5b48a893-be3e-4060-b610-3365757a2b8c"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:26:01.877644 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:26:01.877625    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5b48a893-be3e-4060-b610-3365757a2b8c-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "5b48a893-be3e-4060-b610-3365757a2b8c" (UID: "5b48a893-be3e-4060-b610-3365757a2b8c"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:26:01.976105 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:26:01.976051    2584 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5b48a893-be3e-4060-b610-3365757a2b8c-openshift-service-ca-bundle\") on node \"ip-10-0-142-4.ec2.internal\" DevicePath \"\""
Sep 17 23:26:01.976105 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:26:01.976074    2584 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5b48a893-be3e-4060-b610-3365757a2b8c-proxy-tls\") on node \"ip-10-0-142-4.ec2.internal\" DevicePath \"\""
Sep 17 23:26:02.168149 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:26:02.168122    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/model-chainer-raw-hpa-d5772-75ffc6d89c-k2m2h"]
Sep 17 23:26:02.171737 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:26:02.171713    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/model-chainer-raw-hpa-d5772-75ffc6d89c-k2m2h"]
Sep 17 23:26:02.763452 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:26:02.763424    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5b48a893-be3e-4060-b610-3365757a2b8c" path="/var/lib/kubelet/pods/5b48a893-be3e-4060-b610-3365757a2b8c/volumes"
Sep 17 23:26:04.844756 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:26:04.844731    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-0a0da-predictor-64d6ff776d-ccvv9"
Sep 17 23:26:04.845314 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:26:04.845286    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-0a0da-predictor-64d6ff776d-ccvv9" podUID="42bb7c6e-5c8f-466d-b6db-d6c2e6b65e68" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Sep 17 23:26:04.845505 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:26:04.845475    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-0a0da-predictor-64d6ff776d-ccvv9" podUID="42bb7c6e-5c8f-466d-b6db-d6c2e6b65e68" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 23:26:14.845501 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:26:14.845462    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-0a0da-predictor-64d6ff776d-ccvv9" podUID="42bb7c6e-5c8f-466d-b6db-d6c2e6b65e68" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Sep 17 23:26:14.845942 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:26:14.845920    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-0a0da-predictor-64d6ff776d-ccvv9" podUID="42bb7c6e-5c8f-466d-b6db-d6c2e6b65e68" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 23:26:24.845725 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:26:24.845669    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-0a0da-predictor-64d6ff776d-ccvv9" podUID="42bb7c6e-5c8f-466d-b6db-d6c2e6b65e68" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Sep 17 23:26:24.846216 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:26:24.846168    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-0a0da-predictor-64d6ff776d-ccvv9" podUID="42bb7c6e-5c8f-466d-b6db-d6c2e6b65e68" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 23:26:34.845685 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:26:34.845584    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-0a0da-predictor-64d6ff776d-ccvv9" podUID="42bb7c6e-5c8f-466d-b6db-d6c2e6b65e68" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Sep 17 23:26:34.846205 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:26:34.846043    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-0a0da-predictor-64d6ff776d-ccvv9" podUID="42bb7c6e-5c8f-466d-b6db-d6c2e6b65e68" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 23:26:44.845299 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:26:44.845252    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-0a0da-predictor-64d6ff776d-ccvv9" podUID="42bb7c6e-5c8f-466d-b6db-d6c2e6b65e68" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Sep 17 23:26:44.845749 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:26:44.845721    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-0a0da-predictor-64d6ff776d-ccvv9" podUID="42bb7c6e-5c8f-466d-b6db-d6c2e6b65e68" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 23:26:46.816077 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:26:46.816041    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:26:47.161291 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:26:47.161208    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:26:54.845792 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:26:54.845747    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-0a0da-predictor-64d6ff776d-ccvv9" podUID="42bb7c6e-5c8f-466d-b6db-d6c2e6b65e68" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Sep 17 23:26:54.846294 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:26:54.846271    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-0a0da-predictor-64d6ff776d-ccvv9" podUID="42bb7c6e-5c8f-466d-b6db-d6c2e6b65e68" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 23:27:04.845496 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:27:04.845450    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-0a0da-predictor-64d6ff776d-ccvv9" podUID="42bb7c6e-5c8f-466d-b6db-d6c2e6b65e68" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Sep 17 23:27:04.845983 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:27:04.845886    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-0a0da-predictor-64d6ff776d-ccvv9" podUID="42bb7c6e-5c8f-466d-b6db-d6c2e6b65e68" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 23:27:04.935956 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:27:04.935919    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:27:06.841846 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:27:06.841808    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:27:10.161767 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:27:10.161738    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:27:10.474445 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:27:10.474358    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:27:14.846217 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:27:14.846182    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-0a0da-predictor-64d6ff776d-ccvv9"
Sep 17 23:27:14.846659 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:27:14.846530    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-0a0da-predictor-64d6ff776d-ccvv9"
Sep 17 23:27:15.161366 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:27:15.161257    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:27:20.331512 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:27:20.331480    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:27:20.401580 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:27:20.401542    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:27:27.037580 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:27:27.037545    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-0a0da-predictor-64d6ff776d-ccvv9"]
Sep 17 23:27:27.038049 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:27:27.037916    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-0a0da-predictor-64d6ff776d-ccvv9" podUID="42bb7c6e-5c8f-466d-b6db-d6c2e6b65e68" containerName="agent" containerID="cri-o://46921f616f8e8fc0e4a5802ca18a2f9b8d9516260fd3d1ca5b40a79bc620a9e3" gracePeriod=30
Sep 17 23:27:27.038049 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:27:27.037954    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-0a0da-predictor-64d6ff776d-ccvv9" podUID="42bb7c6e-5c8f-466d-b6db-d6c2e6b65e68" containerName="kube-rbac-proxy" containerID="cri-o://4beb4c7ba12dbb8275a3b02db7c2564c61fc8eaee5b48f582a7af6fc8df2de10" gracePeriod=30
Sep 17 23:27:27.038233 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:27:27.038179    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-0a0da-predictor-64d6ff776d-ccvv9" podUID="42bb7c6e-5c8f-466d-b6db-d6c2e6b65e68" containerName="kserve-container" containerID="cri-o://5fabeefb40484c77f2cf6c637dd0c949116d093312844bfb10481a169b9a997a" gracePeriod=30
Sep 17 23:27:27.072057 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:27:27.072026    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:27:27.149037 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:27:27.148986    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:27:27.197243 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:27:27.197213    2584 generic.go:356] "Generic (PLEG): container finished" podID="42bb7c6e-5c8f-466d-b6db-d6c2e6b65e68" containerID="4beb4c7ba12dbb8275a3b02db7c2564c61fc8eaee5b48f582a7af6fc8df2de10" exitCode=2
Sep 17 23:27:27.197385 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:27:27.197282    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-0a0da-predictor-64d6ff776d-ccvv9" event={"ID":"42bb7c6e-5c8f-466d-b6db-d6c2e6b65e68","Type":"ContainerDied","Data":"4beb4c7ba12dbb8275a3b02db7c2564c61fc8eaee5b48f582a7af6fc8df2de10"}
Sep 17 23:27:27.240768 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:27:27.240742    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:27:27.503611 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:27:27.503584    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:27:29.840205 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:27:29.840165    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-0a0da-predictor-64d6ff776d-ccvv9" podUID="42bb7c6e-5c8f-466d-b6db-d6c2e6b65e68" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.27:8643/healthz\": dial tcp 10.132.0.27:8643: connect: connection refused"
Sep 17 23:27:31.217516 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:27:31.217482    2584 generic.go:356] "Generic (PLEG): container finished" podID="42bb7c6e-5c8f-466d-b6db-d6c2e6b65e68" containerID="5fabeefb40484c77f2cf6c637dd0c949116d093312844bfb10481a169b9a997a" exitCode=0
Sep 17 23:27:31.217890 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:27:31.217550    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-0a0da-predictor-64d6ff776d-ccvv9" event={"ID":"42bb7c6e-5c8f-466d-b6db-d6c2e6b65e68","Type":"ContainerDied","Data":"5fabeefb40484c77f2cf6c637dd0c949116d093312844bfb10481a169b9a997a"}
Sep 17 23:27:34.840484 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:27:34.840446    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-0a0da-predictor-64d6ff776d-ccvv9" podUID="42bb7c6e-5c8f-466d-b6db-d6c2e6b65e68" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.27:8643/healthz\": dial tcp 10.132.0.27:8643: connect: connection refused"
Sep 17 23:27:34.845780 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:27:34.845756    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-0a0da-predictor-64d6ff776d-ccvv9" podUID="42bb7c6e-5c8f-466d-b6db-d6c2e6b65e68" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Sep 17 23:27:34.846190 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:27:34.846158    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-0a0da-predictor-64d6ff776d-ccvv9" podUID="42bb7c6e-5c8f-466d-b6db-d6c2e6b65e68" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 23:27:39.840620 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:27:39.840583    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-0a0da-predictor-64d6ff776d-ccvv9" podUID="42bb7c6e-5c8f-466d-b6db-d6c2e6b65e68" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.27:8643/healthz\": dial tcp 10.132.0.27:8643: connect: connection refused"
Sep 17 23:27:39.841101 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:27:39.840702    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-0a0da-predictor-64d6ff776d-ccvv9"
Sep 17 23:27:44.840357 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:27:44.840318    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-0a0da-predictor-64d6ff776d-ccvv9" podUID="42bb7c6e-5c8f-466d-b6db-d6c2e6b65e68" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.27:8643/healthz\": dial tcp 10.132.0.27:8643: connect: connection refused"
Sep 17 23:27:44.845718 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:27:44.845678    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-0a0da-predictor-64d6ff776d-ccvv9" podUID="42bb7c6e-5c8f-466d-b6db-d6c2e6b65e68" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Sep 17 23:27:44.846034 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:27:44.845989    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-0a0da-predictor-64d6ff776d-ccvv9" podUID="42bb7c6e-5c8f-466d-b6db-d6c2e6b65e68" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 23:27:49.840789 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:27:49.840741    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-0a0da-predictor-64d6ff776d-ccvv9" podUID="42bb7c6e-5c8f-466d-b6db-d6c2e6b65e68" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.27:8643/healthz\": dial tcp 10.132.0.27:8643: connect: connection refused"
Sep 17 23:27:54.840756 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:27:54.840709    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-0a0da-predictor-64d6ff776d-ccvv9" podUID="42bb7c6e-5c8f-466d-b6db-d6c2e6b65e68" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.27:8643/healthz\": dial tcp 10.132.0.27:8643: connect: connection refused"
Sep 17 23:27:54.846072 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:27:54.846042    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-0a0da-predictor-64d6ff776d-ccvv9" podUID="42bb7c6e-5c8f-466d-b6db-d6c2e6b65e68" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Sep 17 23:27:54.846217 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:27:54.846175    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-0a0da-predictor-64d6ff776d-ccvv9"
Sep 17 23:27:54.846436 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:27:54.846413    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-0a0da-predictor-64d6ff776d-ccvv9" podUID="42bb7c6e-5c8f-466d-b6db-d6c2e6b65e68" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 23:27:54.846543 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:27:54.846523    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-0a0da-predictor-64d6ff776d-ccvv9"
Sep 17 23:27:57.182109 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:27:57.182084    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-0a0da-predictor-64d6ff776d-ccvv9"
Sep 17 23:27:57.298509 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:27:57.298440    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/42bb7c6e-5c8f-466d-b6db-d6c2e6b65e68-proxy-tls\") pod \"42bb7c6e-5c8f-466d-b6db-d6c2e6b65e68\" (UID: \"42bb7c6e-5c8f-466d-b6db-d6c2e6b65e68\") "
Sep 17 23:27:57.298509 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:27:57.298498    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/42bb7c6e-5c8f-466d-b6db-d6c2e6b65e68-kube-rbac-proxy-sar-config\") pod \"42bb7c6e-5c8f-466d-b6db-d6c2e6b65e68\" (UID: \"42bb7c6e-5c8f-466d-b6db-d6c2e6b65e68\") "
Sep 17 23:27:57.298722 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:27:57.298547    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/42bb7c6e-5c8f-466d-b6db-d6c2e6b65e68-kserve-provision-location\") pod \"42bb7c6e-5c8f-466d-b6db-d6c2e6b65e68\" (UID: \"42bb7c6e-5c8f-466d-b6db-d6c2e6b65e68\") "
Sep 17 23:27:57.298722 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:27:57.298564    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gqffc\" (UniqueName: \"kubernetes.io/projected/42bb7c6e-5c8f-466d-b6db-d6c2e6b65e68-kube-api-access-gqffc\") pod \"42bb7c6e-5c8f-466d-b6db-d6c2e6b65e68\" (UID: \"42bb7c6e-5c8f-466d-b6db-d6c2e6b65e68\") "
Sep 17 23:27:57.298722 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:27:57.298595    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/42bb7c6e-5c8f-466d-b6db-d6c2e6b65e68-cabundle-cert\") pod \"42bb7c6e-5c8f-466d-b6db-d6c2e6b65e68\" (UID: \"42bb7c6e-5c8f-466d-b6db-d6c2e6b65e68\") "
Sep 17 23:27:57.298938 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:27:57.298913    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42bb7c6e-5c8f-466d-b6db-d6c2e6b65e68-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "42bb7c6e-5c8f-466d-b6db-d6c2e6b65e68" (UID: "42bb7c6e-5c8f-466d-b6db-d6c2e6b65e68"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:27:57.299035 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:27:57.298879    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42bb7c6e-5c8f-466d-b6db-d6c2e6b65e68-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "42bb7c6e-5c8f-466d-b6db-d6c2e6b65e68" (UID: "42bb7c6e-5c8f-466d-b6db-d6c2e6b65e68"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:27:57.299085 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:27:57.299057    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42bb7c6e-5c8f-466d-b6db-d6c2e6b65e68-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "42bb7c6e-5c8f-466d-b6db-d6c2e6b65e68" (UID: "42bb7c6e-5c8f-466d-b6db-d6c2e6b65e68"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:27:57.300722 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:27:57.300703    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/42bb7c6e-5c8f-466d-b6db-d6c2e6b65e68-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "42bb7c6e-5c8f-466d-b6db-d6c2e6b65e68" (UID: "42bb7c6e-5c8f-466d-b6db-d6c2e6b65e68"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:27:57.300822 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:27:57.300808    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42bb7c6e-5c8f-466d-b6db-d6c2e6b65e68-kube-api-access-gqffc" (OuterVolumeSpecName: "kube-api-access-gqffc") pod "42bb7c6e-5c8f-466d-b6db-d6c2e6b65e68" (UID: "42bb7c6e-5c8f-466d-b6db-d6c2e6b65e68"). InnerVolumeSpecName "kube-api-access-gqffc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:27:57.332333 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:27:57.332302    2584 generic.go:356] "Generic (PLEG): container finished" podID="42bb7c6e-5c8f-466d-b6db-d6c2e6b65e68" containerID="46921f616f8e8fc0e4a5802ca18a2f9b8d9516260fd3d1ca5b40a79bc620a9e3" exitCode=0
Sep 17 23:27:57.332438 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:27:57.332358    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-0a0da-predictor-64d6ff776d-ccvv9" event={"ID":"42bb7c6e-5c8f-466d-b6db-d6c2e6b65e68","Type":"ContainerDied","Data":"46921f616f8e8fc0e4a5802ca18a2f9b8d9516260fd3d1ca5b40a79bc620a9e3"}
Sep 17 23:27:57.332438 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:27:57.332365    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-0a0da-predictor-64d6ff776d-ccvv9"
Sep 17 23:27:57.332552 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:27:57.332450    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-0a0da-predictor-64d6ff776d-ccvv9" event={"ID":"42bb7c6e-5c8f-466d-b6db-d6c2e6b65e68","Type":"ContainerDied","Data":"1b22df02beee00a2fcf37b222acf042eaa789f5607eb579e9c2efdd034dd374d"}
Sep 17 23:27:57.332552 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:27:57.332493    2584 scope.go:117] "RemoveContainer" containerID="46921f616f8e8fc0e4a5802ca18a2f9b8d9516260fd3d1ca5b40a79bc620a9e3"
Sep 17 23:27:57.341124 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:27:57.341101    2584 scope.go:117] "RemoveContainer" containerID="4beb4c7ba12dbb8275a3b02db7c2564c61fc8eaee5b48f582a7af6fc8df2de10"
Sep 17 23:27:57.348286 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:27:57.348270    2584 scope.go:117] "RemoveContainer" containerID="5fabeefb40484c77f2cf6c637dd0c949116d093312844bfb10481a169b9a997a"
Sep 17 23:27:57.355454 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:27:57.355433    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-0a0da-predictor-64d6ff776d-ccvv9"]
Sep 17 23:27:57.356260 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:27:57.356245    2584 scope.go:117] "RemoveContainer" containerID="bfe5a864fcdc035838f1c9ce419b6e9ad400210b07715776795bfeaf833a3342"
Sep 17 23:27:57.356698 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:27:57.356680    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-0a0da-predictor-64d6ff776d-ccvv9"]
Sep 17 23:27:57.363314 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:27:57.363299    2584 scope.go:117] "RemoveContainer" containerID="46921f616f8e8fc0e4a5802ca18a2f9b8d9516260fd3d1ca5b40a79bc620a9e3"
Sep 17 23:27:57.363571 ip-10-0-142-4 kubenswrapper[2584]: E0917 23:27:57.363554    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"46921f616f8e8fc0e4a5802ca18a2f9b8d9516260fd3d1ca5b40a79bc620a9e3\": container with ID starting with 46921f616f8e8fc0e4a5802ca18a2f9b8d9516260fd3d1ca5b40a79bc620a9e3 not found: ID does not exist" containerID="46921f616f8e8fc0e4a5802ca18a2f9b8d9516260fd3d1ca5b40a79bc620a9e3"
Sep 17 23:27:57.363614 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:27:57.363577    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"46921f616f8e8fc0e4a5802ca18a2f9b8d9516260fd3d1ca5b40a79bc620a9e3"} err="failed to get container status \"46921f616f8e8fc0e4a5802ca18a2f9b8d9516260fd3d1ca5b40a79bc620a9e3\": rpc error: code = NotFound desc = could not find container \"46921f616f8e8fc0e4a5802ca18a2f9b8d9516260fd3d1ca5b40a79bc620a9e3\": container with ID starting with 46921f616f8e8fc0e4a5802ca18a2f9b8d9516260fd3d1ca5b40a79bc620a9e3 not found: ID does not exist"
Sep 17 23:27:57.363614 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:27:57.363593    2584 scope.go:117] "RemoveContainer" containerID="4beb4c7ba12dbb8275a3b02db7c2564c61fc8eaee5b48f582a7af6fc8df2de10"
Sep 17 23:27:57.363835 ip-10-0-142-4 kubenswrapper[2584]: E0917 23:27:57.363817    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4beb4c7ba12dbb8275a3b02db7c2564c61fc8eaee5b48f582a7af6fc8df2de10\": container with ID starting with 4beb4c7ba12dbb8275a3b02db7c2564c61fc8eaee5b48f582a7af6fc8df2de10 not found: ID does not exist" containerID="4beb4c7ba12dbb8275a3b02db7c2564c61fc8eaee5b48f582a7af6fc8df2de10"
Sep 17 23:27:57.363900 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:27:57.363841    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4beb4c7ba12dbb8275a3b02db7c2564c61fc8eaee5b48f582a7af6fc8df2de10"} err="failed to get container status \"4beb4c7ba12dbb8275a3b02db7c2564c61fc8eaee5b48f582a7af6fc8df2de10\": rpc error: code = NotFound desc = could not find container \"4beb4c7ba12dbb8275a3b02db7c2564c61fc8eaee5b48f582a7af6fc8df2de10\": container with ID starting with 4beb4c7ba12dbb8275a3b02db7c2564c61fc8eaee5b48f582a7af6fc8df2de10 not found: ID does not exist"
Sep 17 23:27:57.363900 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:27:57.363857    2584 scope.go:117] "RemoveContainer" containerID="5fabeefb40484c77f2cf6c637dd0c949116d093312844bfb10481a169b9a997a"
Sep 17 23:27:57.364128 ip-10-0-142-4 kubenswrapper[2584]: E0917 23:27:57.364111    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5fabeefb40484c77f2cf6c637dd0c949116d093312844bfb10481a169b9a997a\": container with ID starting with 5fabeefb40484c77f2cf6c637dd0c949116d093312844bfb10481a169b9a997a not found: ID does not exist" containerID="5fabeefb40484c77f2cf6c637dd0c949116d093312844bfb10481a169b9a997a"
Sep 17 23:27:57.364173 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:27:57.364132    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5fabeefb40484c77f2cf6c637dd0c949116d093312844bfb10481a169b9a997a"} err="failed to get container status \"5fabeefb40484c77f2cf6c637dd0c949116d093312844bfb10481a169b9a997a\": rpc error: code = NotFound desc = could not find container \"5fabeefb40484c77f2cf6c637dd0c949116d093312844bfb10481a169b9a997a\": container with ID starting with 5fabeefb40484c77f2cf6c637dd0c949116d093312844bfb10481a169b9a997a not found: ID does not exist"
Sep 17 23:27:57.364173 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:27:57.364145    2584 scope.go:117] "RemoveContainer" containerID="bfe5a864fcdc035838f1c9ce419b6e9ad400210b07715776795bfeaf833a3342"
Sep 17 23:27:57.364375 ip-10-0-142-4 kubenswrapper[2584]: E0917 23:27:57.364359    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bfe5a864fcdc035838f1c9ce419b6e9ad400210b07715776795bfeaf833a3342\": container with ID starting with bfe5a864fcdc035838f1c9ce419b6e9ad400210b07715776795bfeaf833a3342 not found: ID does not exist" containerID="bfe5a864fcdc035838f1c9ce419b6e9ad400210b07715776795bfeaf833a3342"
Sep 17 23:27:57.364422 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:27:57.364377    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bfe5a864fcdc035838f1c9ce419b6e9ad400210b07715776795bfeaf833a3342"} err="failed to get container status \"bfe5a864fcdc035838f1c9ce419b6e9ad400210b07715776795bfeaf833a3342\": rpc error: code = NotFound desc = could not find container \"bfe5a864fcdc035838f1c9ce419b6e9ad400210b07715776795bfeaf833a3342\": container with ID starting with bfe5a864fcdc035838f1c9ce419b6e9ad400210b07715776795bfeaf833a3342 not found: ID does not exist"
Sep 17 23:27:57.399748 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:27:57.399727    2584 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/42bb7c6e-5c8f-466d-b6db-d6c2e6b65e68-cabundle-cert\") on node \"ip-10-0-142-4.ec2.internal\" DevicePath \"\""
Sep 17 23:27:57.399748 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:27:57.399748    2584 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/42bb7c6e-5c8f-466d-b6db-d6c2e6b65e68-proxy-tls\") on node \"ip-10-0-142-4.ec2.internal\" DevicePath \"\""
Sep 17 23:27:57.399855 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:27:57.399757    2584 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/42bb7c6e-5c8f-466d-b6db-d6c2e6b65e68-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-4.ec2.internal\" DevicePath \"\""
Sep 17 23:27:57.399855 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:27:57.399766    2584 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/42bb7c6e-5c8f-466d-b6db-d6c2e6b65e68-kserve-provision-location\") on node \"ip-10-0-142-4.ec2.internal\" DevicePath \"\""
Sep 17 23:27:57.399855 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:27:57.399775    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gqffc\" (UniqueName: \"kubernetes.io/projected/42bb7c6e-5c8f-466d-b6db-d6c2e6b65e68-kube-api-access-gqffc\") on node \"ip-10-0-142-4.ec2.internal\" DevicePath \"\""
Sep 17 23:27:58.763262 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:27:58.763184    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="42bb7c6e-5c8f-466d-b6db-d6c2e6b65e68" path="/var/lib/kubelet/pods/42bb7c6e-5c8f-466d-b6db-d6c2e6b65e68/volumes"
Sep 17 23:28:19.876923 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:28:19.876890    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:28:50.636298 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:28:50.636263    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:28:55.659623 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:28:55.659588    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:28:55.941052 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:28:55.940959    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:30:17.275184 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:30:17.275151    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:30:19.708662 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:30:19.708633    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:30:22.465314 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:30:22.465285    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:30:22.551970 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:30:22.551940    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-65f7bb5954-dmgqr"]
Sep 17 23:30:22.552272 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:30:22.552259    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="42bb7c6e-5c8f-466d-b6db-d6c2e6b65e68" containerName="storage-initializer"
Sep 17 23:30:22.552272 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:30:22.552274    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="42bb7c6e-5c8f-466d-b6db-d6c2e6b65e68" containerName="storage-initializer"
Sep 17 23:30:22.552365 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:30:22.552283    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5b48a893-be3e-4060-b610-3365757a2b8c" containerName="model-chainer-raw-hpa-d5772"
Sep 17 23:30:22.552365 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:30:22.552289    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="5b48a893-be3e-4060-b610-3365757a2b8c" containerName="model-chainer-raw-hpa-d5772"
Sep 17 23:30:22.552365 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:30:22.552296    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="42bb7c6e-5c8f-466d-b6db-d6c2e6b65e68" containerName="kserve-container"
Sep 17 23:30:22.552365 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:30:22.552302    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="42bb7c6e-5c8f-466d-b6db-d6c2e6b65e68" containerName="kserve-container"
Sep 17 23:30:22.552365 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:30:22.552310    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="42bb7c6e-5c8f-466d-b6db-d6c2e6b65e68" containerName="kube-rbac-proxy"
Sep 17 23:30:22.552365 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:30:22.552315    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="42bb7c6e-5c8f-466d-b6db-d6c2e6b65e68" containerName="kube-rbac-proxy"
Sep 17 23:30:22.552365 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:30:22.552322    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="42bb7c6e-5c8f-466d-b6db-d6c2e6b65e68" containerName="agent"
Sep 17 23:30:22.552365 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:30:22.552327    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="42bb7c6e-5c8f-466d-b6db-d6c2e6b65e68" containerName="agent"
Sep 17 23:30:22.552636 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:30:22.552379    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="42bb7c6e-5c8f-466d-b6db-d6c2e6b65e68" containerName="kube-rbac-proxy"
Sep 17 23:30:22.552636 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:30:22.552389    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="5b48a893-be3e-4060-b610-3365757a2b8c" containerName="model-chainer-raw-hpa-d5772"
Sep 17 23:30:22.552636 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:30:22.552394    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="42bb7c6e-5c8f-466d-b6db-d6c2e6b65e68" containerName="agent"
Sep 17 23:30:22.552636 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:30:22.552403    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="42bb7c6e-5c8f-466d-b6db-d6c2e6b65e68" containerName="kserve-container"
Sep 17 23:30:22.555338 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:30:22.555323    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-65f7bb5954-dmgqr"
Sep 17 23:30:22.557344 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:30:22.557324    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:30:22.557925 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:30:22.557905    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 23:30:22.558031 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:30:22.557929    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-predictor-serving-cert\""
Sep 17 23:30:22.558031 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:30:22.557933    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 23:30:22.558711 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:30:22.558681    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-f2cbg\""
Sep 17 23:30:22.558804 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:30:22.558729    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 23:30:22.558804 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:30:22.558731    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 23:30:22.558979 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:30:22.558806    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-kube-rbac-proxy-sar-config\""
Sep 17 23:30:22.567711 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:30:22.567687    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-65f7bb5954-dmgqr"]
Sep 17 23:30:22.608747 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:30:22.608718    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/13f44ba0-651e-4c83-9fd4-51cefaf67cbe-kserve-provision-location\") pod \"isvc-canary-create-predictor-65f7bb5954-dmgqr\" (UID: \"13f44ba0-651e-4c83-9fd4-51cefaf67cbe\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-65f7bb5954-dmgqr"
Sep 17 23:30:22.608859 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:30:22.608765    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/13f44ba0-651e-4c83-9fd4-51cefaf67cbe-cabundle-cert\") pod \"isvc-canary-create-predictor-65f7bb5954-dmgqr\" (UID: \"13f44ba0-651e-4c83-9fd4-51cefaf67cbe\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-65f7bb5954-dmgqr"
Sep 17 23:30:22.608859 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:30:22.608824    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l4bpq\" (UniqueName: \"kubernetes.io/projected/13f44ba0-651e-4c83-9fd4-51cefaf67cbe-kube-api-access-l4bpq\") pod \"isvc-canary-create-predictor-65f7bb5954-dmgqr\" (UID: \"13f44ba0-651e-4c83-9fd4-51cefaf67cbe\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-65f7bb5954-dmgqr"
Sep 17 23:30:22.608859 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:30:22.608858    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/13f44ba0-651e-4c83-9fd4-51cefaf67cbe-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-65f7bb5954-dmgqr\" (UID: \"13f44ba0-651e-4c83-9fd4-51cefaf67cbe\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-65f7bb5954-dmgqr"
Sep 17 23:30:22.608999 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:30:22.608884    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/13f44ba0-651e-4c83-9fd4-51cefaf67cbe-proxy-tls\") pod \"isvc-canary-create-predictor-65f7bb5954-dmgqr\" (UID: \"13f44ba0-651e-4c83-9fd4-51cefaf67cbe\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-65f7bb5954-dmgqr"
Sep 17 23:30:22.709940 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:30:22.709908    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/13f44ba0-651e-4c83-9fd4-51cefaf67cbe-kserve-provision-location\") pod \"isvc-canary-create-predictor-65f7bb5954-dmgqr\" (UID: \"13f44ba0-651e-4c83-9fd4-51cefaf67cbe\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-65f7bb5954-dmgqr"
Sep 17 23:30:22.710067 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:30:22.709968    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/13f44ba0-651e-4c83-9fd4-51cefaf67cbe-cabundle-cert\") pod \"isvc-canary-create-predictor-65f7bb5954-dmgqr\" (UID: \"13f44ba0-651e-4c83-9fd4-51cefaf67cbe\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-65f7bb5954-dmgqr"
Sep 17 23:30:22.710067 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:30:22.710037    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l4bpq\" (UniqueName: \"kubernetes.io/projected/13f44ba0-651e-4c83-9fd4-51cefaf67cbe-kube-api-access-l4bpq\") pod \"isvc-canary-create-predictor-65f7bb5954-dmgqr\" (UID: \"13f44ba0-651e-4c83-9fd4-51cefaf67cbe\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-65f7bb5954-dmgqr"
Sep 17 23:30:22.710200 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:30:22.710069    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/13f44ba0-651e-4c83-9fd4-51cefaf67cbe-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-65f7bb5954-dmgqr\" (UID: \"13f44ba0-651e-4c83-9fd4-51cefaf67cbe\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-65f7bb5954-dmgqr"
Sep 17 23:30:22.710200 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:30:22.710102    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/13f44ba0-651e-4c83-9fd4-51cefaf67cbe-proxy-tls\") pod \"isvc-canary-create-predictor-65f7bb5954-dmgqr\" (UID: \"13f44ba0-651e-4c83-9fd4-51cefaf67cbe\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-65f7bb5954-dmgqr"
Sep 17 23:30:22.710304 ip-10-0-142-4 kubenswrapper[2584]: E0917 23:30:22.710222    2584 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-create-predictor-serving-cert: secret "isvc-canary-create-predictor-serving-cert" not found
Sep 17 23:30:22.710304 ip-10-0-142-4 kubenswrapper[2584]: E0917 23:30:22.710282    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/13f44ba0-651e-4c83-9fd4-51cefaf67cbe-proxy-tls podName:13f44ba0-651e-4c83-9fd4-51cefaf67cbe nodeName:}" failed. No retries permitted until 2026-09-17 23:30:23.210260699 +0000 UTC m=+1206.575920609 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/13f44ba0-651e-4c83-9fd4-51cefaf67cbe-proxy-tls") pod "isvc-canary-create-predictor-65f7bb5954-dmgqr" (UID: "13f44ba0-651e-4c83-9fd4-51cefaf67cbe") : secret "isvc-canary-create-predictor-serving-cert" not found
Sep 17 23:30:22.710391 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:30:22.710317    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/13f44ba0-651e-4c83-9fd4-51cefaf67cbe-kserve-provision-location\") pod \"isvc-canary-create-predictor-65f7bb5954-dmgqr\" (UID: \"13f44ba0-651e-4c83-9fd4-51cefaf67cbe\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-65f7bb5954-dmgqr"
Sep 17 23:30:22.710653 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:30:22.710636    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/13f44ba0-651e-4c83-9fd4-51cefaf67cbe-cabundle-cert\") pod \"isvc-canary-create-predictor-65f7bb5954-dmgqr\" (UID: \"13f44ba0-651e-4c83-9fd4-51cefaf67cbe\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-65f7bb5954-dmgqr"
Sep 17 23:30:22.710653 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:30:22.710645    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/13f44ba0-651e-4c83-9fd4-51cefaf67cbe-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-65f7bb5954-dmgqr\" (UID: \"13f44ba0-651e-4c83-9fd4-51cefaf67cbe\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-65f7bb5954-dmgqr"
Sep 17 23:30:22.724330 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:30:22.724290    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l4bpq\" (UniqueName: \"kubernetes.io/projected/13f44ba0-651e-4c83-9fd4-51cefaf67cbe-kube-api-access-l4bpq\") pod \"isvc-canary-create-predictor-65f7bb5954-dmgqr\" (UID: \"13f44ba0-651e-4c83-9fd4-51cefaf67cbe\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-65f7bb5954-dmgqr"
Sep 17 23:30:22.900725 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:30:22.900684    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:30:23.154016 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:30:23.153987    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:30:23.212496 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:30:23.212465    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/13f44ba0-651e-4c83-9fd4-51cefaf67cbe-proxy-tls\") pod \"isvc-canary-create-predictor-65f7bb5954-dmgqr\" (UID: \"13f44ba0-651e-4c83-9fd4-51cefaf67cbe\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-65f7bb5954-dmgqr"
Sep 17 23:30:23.214932 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:30:23.214906    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/13f44ba0-651e-4c83-9fd4-51cefaf67cbe-proxy-tls\") pod \"isvc-canary-create-predictor-65f7bb5954-dmgqr\" (UID: \"13f44ba0-651e-4c83-9fd4-51cefaf67cbe\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-65f7bb5954-dmgqr"
Sep 17 23:30:23.467058 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:30:23.466970    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-65f7bb5954-dmgqr"
Sep 17 23:30:23.739764 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:30:23.739696    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-65f7bb5954-dmgqr"]
Sep 17 23:30:23.743393 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:30:23.743369    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13f44ba0_651e_4c83_9fd4_51cefaf67cbe.slice/crio-6fe4125b2b2ceeaaa4f21a7ad2e621e4708b4e1a02a64d8731951b2d0010512d WatchSource:0}: Error finding container 6fe4125b2b2ceeaaa4f21a7ad2e621e4708b4e1a02a64d8731951b2d0010512d: Status 404 returned error can't find the container with id 6fe4125b2b2ceeaaa4f21a7ad2e621e4708b4e1a02a64d8731951b2d0010512d
Sep 17 23:30:23.902020 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:30:23.901980    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-65f7bb5954-dmgqr" event={"ID":"13f44ba0-651e-4c83-9fd4-51cefaf67cbe","Type":"ContainerStarted","Data":"6861f9be3be3beefd9371f00b1bba7d8d8b5ac12d068e5486f74ba62648506a3"}
Sep 17 23:30:23.902190 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:30:23.902068    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-65f7bb5954-dmgqr" event={"ID":"13f44ba0-651e-4c83-9fd4-51cefaf67cbe","Type":"ContainerStarted","Data":"6fe4125b2b2ceeaaa4f21a7ad2e621e4708b4e1a02a64d8731951b2d0010512d"}
Sep 17 23:30:25.836566 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:30:25.836535    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:30:27.919921 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:30:27.919890    2584 generic.go:356] "Generic (PLEG): container finished" podID="13f44ba0-651e-4c83-9fd4-51cefaf67cbe" containerID="6861f9be3be3beefd9371f00b1bba7d8d8b5ac12d068e5486f74ba62648506a3" exitCode=0
Sep 17 23:30:27.920313 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:30:27.919968    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-65f7bb5954-dmgqr" event={"ID":"13f44ba0-651e-4c83-9fd4-51cefaf67cbe","Type":"ContainerDied","Data":"6861f9be3be3beefd9371f00b1bba7d8d8b5ac12d068e5486f74ba62648506a3"}
Sep 17 23:30:28.926359 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:30:28.926323    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-65f7bb5954-dmgqr" event={"ID":"13f44ba0-651e-4c83-9fd4-51cefaf67cbe","Type":"ContainerStarted","Data":"b115a6b960f60b369ea8d37d8d7e6d476e6e005541c0625b47c275aefe9ad6d0"}
Sep 17 23:30:28.926837 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:30:28.926393    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-65f7bb5954-dmgqr" event={"ID":"13f44ba0-651e-4c83-9fd4-51cefaf67cbe","Type":"ContainerStarted","Data":"790334cec30d1421d9a30364636644d1ca2ff56fea6e81dad3efdb1015f0cd5a"}
Sep 17 23:30:28.926837 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:30:28.926592    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-65f7bb5954-dmgqr"
Sep 17 23:30:28.950043 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:30:28.949976    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-65f7bb5954-dmgqr" podStartSLOduration=3.262422162 podStartE2EDuration="6.949960942s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.68753878s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:30:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:30:28.948975251 +0000 UTC m=+1212.314635167" watchObservedRunningTime="2026-09-17 23:30:28.949960942 +0000 UTC m=+1212.315620915"
Sep 17 23:30:29.930784 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:30:29.930750    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-65f7bb5954-dmgqr"
Sep 17 23:30:29.932114 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:30:29.932088    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-65f7bb5954-dmgqr" podUID="13f44ba0-651e-4c83-9fd4-51cefaf67cbe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused"
Sep 17 23:30:30.934066 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:30:30.934023    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-65f7bb5954-dmgqr" podUID="13f44ba0-651e-4c83-9fd4-51cefaf67cbe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused"
Sep 17 23:30:35.938203 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:30:35.938174    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-65f7bb5954-dmgqr"
Sep 17 23:30:35.938687 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:30:35.938613    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-65f7bb5954-dmgqr" podUID="13f44ba0-651e-4c83-9fd4-51cefaf67cbe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused"
Sep 17 23:30:45.939246 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:30:45.939210    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-65f7bb5954-dmgqr" podUID="13f44ba0-651e-4c83-9fd4-51cefaf67cbe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused"
Sep 17 23:30:55.938928 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:30:55.938885    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-65f7bb5954-dmgqr" podUID="13f44ba0-651e-4c83-9fd4-51cefaf67cbe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused"
Sep 17 23:31:05.939627 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:31:05.939544    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-65f7bb5954-dmgqr" podUID="13f44ba0-651e-4c83-9fd4-51cefaf67cbe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused"
Sep 17 23:31:12.873308 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:31:12.873266    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:31:15.939196 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:31:15.939159    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-65f7bb5954-dmgqr" podUID="13f44ba0-651e-4c83-9fd4-51cefaf67cbe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused"
Sep 17 23:31:25.938638 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:31:25.938590    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-65f7bb5954-dmgqr" podUID="13f44ba0-651e-4c83-9fd4-51cefaf67cbe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused"
Sep 17 23:31:35.940095 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:31:35.940069    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-65f7bb5954-dmgqr"
Sep 17 23:31:44.193752 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:31:44.193715    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:31:52.548891 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:31:52.548857    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-65f7bb5954-dmgqr"]
Sep 17 23:31:52.549312 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:31:52.549174    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-65f7bb5954-dmgqr" podUID="13f44ba0-651e-4c83-9fd4-51cefaf67cbe" containerName="kserve-container" containerID="cri-o://790334cec30d1421d9a30364636644d1ca2ff56fea6e81dad3efdb1015f0cd5a" gracePeriod=30
Sep 17 23:31:52.549312 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:31:52.549216    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-65f7bb5954-dmgqr" podUID="13f44ba0-651e-4c83-9fd4-51cefaf67cbe" containerName="kube-rbac-proxy" containerID="cri-o://b115a6b960f60b369ea8d37d8d7e6d476e6e005541c0625b47c275aefe9ad6d0" gracePeriod=30
Sep 17 23:31:52.571673 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:31:52.571644    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:31:52.667132 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:31:52.667092    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:31:53.253919 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:31:53.253880    2584 generic.go:356] "Generic (PLEG): container finished" podID="13f44ba0-651e-4c83-9fd4-51cefaf67cbe" containerID="b115a6b960f60b369ea8d37d8d7e6d476e6e005541c0625b47c275aefe9ad6d0" exitCode=2
Sep 17 23:31:53.254122 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:31:53.253938    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-65f7bb5954-dmgqr" event={"ID":"13f44ba0-651e-4c83-9fd4-51cefaf67cbe","Type":"ContainerDied","Data":"b115a6b960f60b369ea8d37d8d7e6d476e6e005541c0625b47c275aefe9ad6d0"}
Sep 17 23:31:55.935045 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:31:55.934983    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-65f7bb5954-dmgqr" podUID="13f44ba0-651e-4c83-9fd4-51cefaf67cbe" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.28:8643/healthz\": dial tcp 10.132.0.28:8643: connect: connection refused"
Sep 17 23:31:55.939318 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:31:55.939277    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-65f7bb5954-dmgqr" podUID="13f44ba0-651e-4c83-9fd4-51cefaf67cbe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused"
Sep 17 23:31:56.781995 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:31:56.781963    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-65f7bb5954-dmgqr"
Sep 17 23:31:56.922257 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:31:56.922178    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/13f44ba0-651e-4c83-9fd4-51cefaf67cbe-proxy-tls\") pod \"13f44ba0-651e-4c83-9fd4-51cefaf67cbe\" (UID: \"13f44ba0-651e-4c83-9fd4-51cefaf67cbe\") "
Sep 17 23:31:56.922257 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:31:56.922223    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l4bpq\" (UniqueName: \"kubernetes.io/projected/13f44ba0-651e-4c83-9fd4-51cefaf67cbe-kube-api-access-l4bpq\") pod \"13f44ba0-651e-4c83-9fd4-51cefaf67cbe\" (UID: \"13f44ba0-651e-4c83-9fd4-51cefaf67cbe\") "
Sep 17 23:31:56.922475 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:31:56.922301    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/13f44ba0-651e-4c83-9fd4-51cefaf67cbe-cabundle-cert\") pod \"13f44ba0-651e-4c83-9fd4-51cefaf67cbe\" (UID: \"13f44ba0-651e-4c83-9fd4-51cefaf67cbe\") "
Sep 17 23:31:56.922475 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:31:56.922319    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/13f44ba0-651e-4c83-9fd4-51cefaf67cbe-kserve-provision-location\") pod \"13f44ba0-651e-4c83-9fd4-51cefaf67cbe\" (UID: \"13f44ba0-651e-4c83-9fd4-51cefaf67cbe\") "
Sep 17 23:31:56.922475 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:31:56.922350    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/13f44ba0-651e-4c83-9fd4-51cefaf67cbe-kube-rbac-proxy-sar-config\") pod \"13f44ba0-651e-4c83-9fd4-51cefaf67cbe\" (UID: \"13f44ba0-651e-4c83-9fd4-51cefaf67cbe\") "
Sep 17 23:31:56.922665 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:31:56.922632    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13f44ba0-651e-4c83-9fd4-51cefaf67cbe-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "13f44ba0-651e-4c83-9fd4-51cefaf67cbe" (UID: "13f44ba0-651e-4c83-9fd4-51cefaf67cbe"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:31:56.922719 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:31:56.922683    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13f44ba0-651e-4c83-9fd4-51cefaf67cbe-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "13f44ba0-651e-4c83-9fd4-51cefaf67cbe" (UID: "13f44ba0-651e-4c83-9fd4-51cefaf67cbe"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:31:56.922758 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:31:56.922725    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13f44ba0-651e-4c83-9fd4-51cefaf67cbe-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "13f44ba0-651e-4c83-9fd4-51cefaf67cbe" (UID: "13f44ba0-651e-4c83-9fd4-51cefaf67cbe"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:31:56.924373 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:31:56.924350    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13f44ba0-651e-4c83-9fd4-51cefaf67cbe-kube-api-access-l4bpq" (OuterVolumeSpecName: "kube-api-access-l4bpq") pod "13f44ba0-651e-4c83-9fd4-51cefaf67cbe" (UID: "13f44ba0-651e-4c83-9fd4-51cefaf67cbe"). InnerVolumeSpecName "kube-api-access-l4bpq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:31:56.924486 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:31:56.924419    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/13f44ba0-651e-4c83-9fd4-51cefaf67cbe-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "13f44ba0-651e-4c83-9fd4-51cefaf67cbe" (UID: "13f44ba0-651e-4c83-9fd4-51cefaf67cbe"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:31:57.023698 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:31:57.023666    2584 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/13f44ba0-651e-4c83-9fd4-51cefaf67cbe-cabundle-cert\") on node \"ip-10-0-142-4.ec2.internal\" DevicePath \"\""
Sep 17 23:31:57.024083 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:31:57.023713    2584 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/13f44ba0-651e-4c83-9fd4-51cefaf67cbe-kserve-provision-location\") on node \"ip-10-0-142-4.ec2.internal\" DevicePath \"\""
Sep 17 23:31:57.024083 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:31:57.023723    2584 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/13f44ba0-651e-4c83-9fd4-51cefaf67cbe-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-4.ec2.internal\" DevicePath \"\""
Sep 17 23:31:57.024083 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:31:57.023732    2584 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/13f44ba0-651e-4c83-9fd4-51cefaf67cbe-proxy-tls\") on node \"ip-10-0-142-4.ec2.internal\" DevicePath \"\""
Sep 17 23:31:57.024083 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:31:57.023740    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l4bpq\" (UniqueName: \"kubernetes.io/projected/13f44ba0-651e-4c83-9fd4-51cefaf67cbe-kube-api-access-l4bpq\") on node \"ip-10-0-142-4.ec2.internal\" DevicePath \"\""
Sep 17 23:31:57.271880 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:31:57.271846    2584 generic.go:356] "Generic (PLEG): container finished" podID="13f44ba0-651e-4c83-9fd4-51cefaf67cbe" containerID="790334cec30d1421d9a30364636644d1ca2ff56fea6e81dad3efdb1015f0cd5a" exitCode=0
Sep 17 23:31:57.272097 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:31:57.271891    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-65f7bb5954-dmgqr" event={"ID":"13f44ba0-651e-4c83-9fd4-51cefaf67cbe","Type":"ContainerDied","Data":"790334cec30d1421d9a30364636644d1ca2ff56fea6e81dad3efdb1015f0cd5a"}
Sep 17 23:31:57.272097 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:31:57.271918    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-65f7bb5954-dmgqr"
Sep 17 23:31:57.272097 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:31:57.271937    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-65f7bb5954-dmgqr" event={"ID":"13f44ba0-651e-4c83-9fd4-51cefaf67cbe","Type":"ContainerDied","Data":"6fe4125b2b2ceeaaa4f21a7ad2e621e4708b4e1a02a64d8731951b2d0010512d"}
Sep 17 23:31:57.272097 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:31:57.271963    2584 scope.go:117] "RemoveContainer" containerID="b115a6b960f60b369ea8d37d8d7e6d476e6e005541c0625b47c275aefe9ad6d0"
Sep 17 23:31:57.280445 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:31:57.280429    2584 scope.go:117] "RemoveContainer" containerID="790334cec30d1421d9a30364636644d1ca2ff56fea6e81dad3efdb1015f0cd5a"
Sep 17 23:31:57.287886 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:31:57.287870    2584 scope.go:117] "RemoveContainer" containerID="6861f9be3be3beefd9371f00b1bba7d8d8b5ac12d068e5486f74ba62648506a3"
Sep 17 23:31:57.294972 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:31:57.294944    2584 scope.go:117] "RemoveContainer" containerID="b115a6b960f60b369ea8d37d8d7e6d476e6e005541c0625b47c275aefe9ad6d0"
Sep 17 23:31:57.295271 ip-10-0-142-4 kubenswrapper[2584]: E0917 23:31:57.295243    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b115a6b960f60b369ea8d37d8d7e6d476e6e005541c0625b47c275aefe9ad6d0\": container with ID starting with b115a6b960f60b369ea8d37d8d7e6d476e6e005541c0625b47c275aefe9ad6d0 not found: ID does not exist" containerID="b115a6b960f60b369ea8d37d8d7e6d476e6e005541c0625b47c275aefe9ad6d0"
Sep 17 23:31:57.295354 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:31:57.295273    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b115a6b960f60b369ea8d37d8d7e6d476e6e005541c0625b47c275aefe9ad6d0"} err="failed to get container status \"b115a6b960f60b369ea8d37d8d7e6d476e6e005541c0625b47c275aefe9ad6d0\": rpc error: code = NotFound desc = could not find container \"b115a6b960f60b369ea8d37d8d7e6d476e6e005541c0625b47c275aefe9ad6d0\": container with ID starting with b115a6b960f60b369ea8d37d8d7e6d476e6e005541c0625b47c275aefe9ad6d0 not found: ID does not exist"
Sep 17 23:31:57.295354 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:31:57.295298    2584 scope.go:117] "RemoveContainer" containerID="790334cec30d1421d9a30364636644d1ca2ff56fea6e81dad3efdb1015f0cd5a"
Sep 17 23:31:57.295750 ip-10-0-142-4 kubenswrapper[2584]: E0917 23:31:57.295600    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"790334cec30d1421d9a30364636644d1ca2ff56fea6e81dad3efdb1015f0cd5a\": container with ID starting with 790334cec30d1421d9a30364636644d1ca2ff56fea6e81dad3efdb1015f0cd5a not found: ID does not exist" containerID="790334cec30d1421d9a30364636644d1ca2ff56fea6e81dad3efdb1015f0cd5a"
Sep 17 23:31:57.295866 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:31:57.295778    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"790334cec30d1421d9a30364636644d1ca2ff56fea6e81dad3efdb1015f0cd5a"} err="failed to get container status \"790334cec30d1421d9a30364636644d1ca2ff56fea6e81dad3efdb1015f0cd5a\": rpc error: code = NotFound desc = could not find container \"790334cec30d1421d9a30364636644d1ca2ff56fea6e81dad3efdb1015f0cd5a\": container with ID starting with 790334cec30d1421d9a30364636644d1ca2ff56fea6e81dad3efdb1015f0cd5a not found: ID does not exist"
Sep 17 23:31:57.295866 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:31:57.295804    2584 scope.go:117] "RemoveContainer" containerID="6861f9be3be3beefd9371f00b1bba7d8d8b5ac12d068e5486f74ba62648506a3"
Sep 17 23:31:57.296478 ip-10-0-142-4 kubenswrapper[2584]: E0917 23:31:57.296459    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6861f9be3be3beefd9371f00b1bba7d8d8b5ac12d068e5486f74ba62648506a3\": container with ID starting with 6861f9be3be3beefd9371f00b1bba7d8d8b5ac12d068e5486f74ba62648506a3 not found: ID does not exist" containerID="6861f9be3be3beefd9371f00b1bba7d8d8b5ac12d068e5486f74ba62648506a3"
Sep 17 23:31:57.296555 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:31:57.296483    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6861f9be3be3beefd9371f00b1bba7d8d8b5ac12d068e5486f74ba62648506a3"} err="failed to get container status \"6861f9be3be3beefd9371f00b1bba7d8d8b5ac12d068e5486f74ba62648506a3\": rpc error: code = NotFound desc = could not find container \"6861f9be3be3beefd9371f00b1bba7d8d8b5ac12d068e5486f74ba62648506a3\": container with ID starting with 6861f9be3be3beefd9371f00b1bba7d8d8b5ac12d068e5486f74ba62648506a3 not found: ID does not exist"
Sep 17 23:31:57.298547 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:31:57.298524    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-65f7bb5954-dmgqr"]
Sep 17 23:31:57.299811 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:31:57.299793    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-65f7bb5954-dmgqr"]
Sep 17 23:31:57.843986 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:31:57.843953    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:31:57.923209 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:31:57.923179    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:31:57.974868 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:31:57.974836    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-fc78f5cb6-49w9v"]
Sep 17 23:31:57.975169 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:31:57.975157    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="13f44ba0-651e-4c83-9fd4-51cefaf67cbe" containerName="storage-initializer"
Sep 17 23:31:57.975217 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:31:57.975172    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="13f44ba0-651e-4c83-9fd4-51cefaf67cbe" containerName="storage-initializer"
Sep 17 23:31:57.975217 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:31:57.975182    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="13f44ba0-651e-4c83-9fd4-51cefaf67cbe" containerName="kube-rbac-proxy"
Sep 17 23:31:57.975217 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:31:57.975190    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="13f44ba0-651e-4c83-9fd4-51cefaf67cbe" containerName="kube-rbac-proxy"
Sep 17 23:31:57.975217 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:31:57.975199    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="13f44ba0-651e-4c83-9fd4-51cefaf67cbe" containerName="kserve-container"
Sep 17 23:31:57.975217 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:31:57.975204    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="13f44ba0-651e-4c83-9fd4-51cefaf67cbe" containerName="kserve-container"
Sep 17 23:31:57.975375 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:31:57.975259    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="13f44ba0-651e-4c83-9fd4-51cefaf67cbe" containerName="kserve-container"
Sep 17 23:31:57.975375 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:31:57.975269    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="13f44ba0-651e-4c83-9fd4-51cefaf67cbe" containerName="kube-rbac-proxy"
Sep 17 23:31:57.979651 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:31:57.979632    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-fc78f5cb6-49w9v"
Sep 17 23:31:57.982245 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:31:57.982219    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 23:31:57.982380 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:31:57.982244    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 23:31:57.982380 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:31:57.982244    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-kube-rbac-proxy-sar-config\""
Sep 17 23:31:57.982380 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:31:57.982332    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 23:31:57.982380 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:31:57.982332    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 23:31:57.982589 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:31:57.982332    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-predictor-serving-cert\""
Sep 17 23:31:57.983329 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:31:57.983311    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-f2cbg\""
Sep 17 23:31:57.990563 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:31:57.990542    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-fc78f5cb6-49w9v"]
Sep 17 23:31:58.118776 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:31:58.118694    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:31:58.132849 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:31:58.132823    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/eebbc4b3-1248-4ef1-8c34-e2ca6008e746-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-fc78f5cb6-49w9v\" (UID: \"eebbc4b3-1248-4ef1-8c34-e2ca6008e746\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-fc78f5cb6-49w9v"
Sep 17 23:31:58.132928 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:31:58.132872    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/eebbc4b3-1248-4ef1-8c34-e2ca6008e746-kserve-provision-location\") pod \"isvc-canary-promote-predictor-fc78f5cb6-49w9v\" (UID: \"eebbc4b3-1248-4ef1-8c34-e2ca6008e746\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-fc78f5cb6-49w9v"
Sep 17 23:31:58.132928 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:31:58.132900    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/eebbc4b3-1248-4ef1-8c34-e2ca6008e746-proxy-tls\") pod \"isvc-canary-promote-predictor-fc78f5cb6-49w9v\" (UID: \"eebbc4b3-1248-4ef1-8c34-e2ca6008e746\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-fc78f5cb6-49w9v"
Sep 17 23:31:58.133030 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:31:58.132939    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ksv4d\" (UniqueName: \"kubernetes.io/projected/eebbc4b3-1248-4ef1-8c34-e2ca6008e746-kube-api-access-ksv4d\") pod \"isvc-canary-promote-predictor-fc78f5cb6-49w9v\" (UID: \"eebbc4b3-1248-4ef1-8c34-e2ca6008e746\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-fc78f5cb6-49w9v"
Sep 17 23:31:58.133030 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:31:58.132978    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/eebbc4b3-1248-4ef1-8c34-e2ca6008e746-cabundle-cert\") pod \"isvc-canary-promote-predictor-fc78f5cb6-49w9v\" (UID: \"eebbc4b3-1248-4ef1-8c34-e2ca6008e746\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-fc78f5cb6-49w9v"
Sep 17 23:31:58.233931 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:31:58.233890    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/eebbc4b3-1248-4ef1-8c34-e2ca6008e746-kserve-provision-location\") pod \"isvc-canary-promote-predictor-fc78f5cb6-49w9v\" (UID: \"eebbc4b3-1248-4ef1-8c34-e2ca6008e746\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-fc78f5cb6-49w9v"
Sep 17 23:31:58.233931 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:31:58.233932    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/eebbc4b3-1248-4ef1-8c34-e2ca6008e746-proxy-tls\") pod \"isvc-canary-promote-predictor-fc78f5cb6-49w9v\" (UID: \"eebbc4b3-1248-4ef1-8c34-e2ca6008e746\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-fc78f5cb6-49w9v"
Sep 17 23:31:58.234159 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:31:58.233958    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ksv4d\" (UniqueName: \"kubernetes.io/projected/eebbc4b3-1248-4ef1-8c34-e2ca6008e746-kube-api-access-ksv4d\") pod \"isvc-canary-promote-predictor-fc78f5cb6-49w9v\" (UID: \"eebbc4b3-1248-4ef1-8c34-e2ca6008e746\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-fc78f5cb6-49w9v"
Sep 17 23:31:58.234159 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:31:58.233988    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/eebbc4b3-1248-4ef1-8c34-e2ca6008e746-cabundle-cert\") pod \"isvc-canary-promote-predictor-fc78f5cb6-49w9v\" (UID: \"eebbc4b3-1248-4ef1-8c34-e2ca6008e746\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-fc78f5cb6-49w9v"
Sep 17 23:31:58.234159 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:31:58.234059    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/eebbc4b3-1248-4ef1-8c34-e2ca6008e746-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-fc78f5cb6-49w9v\" (UID: \"eebbc4b3-1248-4ef1-8c34-e2ca6008e746\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-fc78f5cb6-49w9v"
Sep 17 23:31:58.234427 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:31:58.234409    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/eebbc4b3-1248-4ef1-8c34-e2ca6008e746-kserve-provision-location\") pod \"isvc-canary-promote-predictor-fc78f5cb6-49w9v\" (UID: \"eebbc4b3-1248-4ef1-8c34-e2ca6008e746\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-fc78f5cb6-49w9v"
Sep 17 23:31:58.234736 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:31:58.234719    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/eebbc4b3-1248-4ef1-8c34-e2ca6008e746-cabundle-cert\") pod \"isvc-canary-promote-predictor-fc78f5cb6-49w9v\" (UID: \"eebbc4b3-1248-4ef1-8c34-e2ca6008e746\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-fc78f5cb6-49w9v"
Sep 17 23:31:58.234776 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:31:58.234757    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/eebbc4b3-1248-4ef1-8c34-e2ca6008e746-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-fc78f5cb6-49w9v\" (UID: \"eebbc4b3-1248-4ef1-8c34-e2ca6008e746\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-fc78f5cb6-49w9v"
Sep 17 23:31:58.236829 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:31:58.236813    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/eebbc4b3-1248-4ef1-8c34-e2ca6008e746-proxy-tls\") pod \"isvc-canary-promote-predictor-fc78f5cb6-49w9v\" (UID: \"eebbc4b3-1248-4ef1-8c34-e2ca6008e746\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-fc78f5cb6-49w9v"
Sep 17 23:31:58.243101 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:31:58.243080    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ksv4d\" (UniqueName: \"kubernetes.io/projected/eebbc4b3-1248-4ef1-8c34-e2ca6008e746-kube-api-access-ksv4d\") pod \"isvc-canary-promote-predictor-fc78f5cb6-49w9v\" (UID: \"eebbc4b3-1248-4ef1-8c34-e2ca6008e746\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-fc78f5cb6-49w9v"
Sep 17 23:31:58.289539 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:31:58.289510    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-fc78f5cb6-49w9v"
Sep 17 23:31:58.561146 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:31:58.561111    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-fc78f5cb6-49w9v"]
Sep 17 23:31:58.563974 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:31:58.563949    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeebbc4b3_1248_4ef1_8c34_e2ca6008e746.slice/crio-f49d7e4fc4a75c0963a7f714bab9221f498a6a3cd58c6c49d355e2907a574fa7 WatchSource:0}: Error finding container f49d7e4fc4a75c0963a7f714bab9221f498a6a3cd58c6c49d355e2907a574fa7: Status 404 returned error can't find the container with id f49d7e4fc4a75c0963a7f714bab9221f498a6a3cd58c6c49d355e2907a574fa7
Sep 17 23:31:58.763814 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:31:58.763776    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="13f44ba0-651e-4c83-9fd4-51cefaf67cbe" path="/var/lib/kubelet/pods/13f44ba0-651e-4c83-9fd4-51cefaf67cbe/volumes"
Sep 17 23:31:59.284035 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:31:59.283985    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-fc78f5cb6-49w9v" event={"ID":"eebbc4b3-1248-4ef1-8c34-e2ca6008e746","Type":"ContainerStarted","Data":"e246eae5f8fb1fb74989d74a43e2b0efdd97da236ac69204b198ce15ab81ae68"}
Sep 17 23:31:59.284412 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:31:59.284070    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-fc78f5cb6-49w9v" event={"ID":"eebbc4b3-1248-4ef1-8c34-e2ca6008e746","Type":"ContainerStarted","Data":"f49d7e4fc4a75c0963a7f714bab9221f498a6a3cd58c6c49d355e2907a574fa7"}
Sep 17 23:32:03.304109 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:32:03.304067    2584 generic.go:356] "Generic (PLEG): container finished" podID="eebbc4b3-1248-4ef1-8c34-e2ca6008e746" containerID="e246eae5f8fb1fb74989d74a43e2b0efdd97da236ac69204b198ce15ab81ae68" exitCode=0
Sep 17 23:32:03.304542 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:32:03.304146    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-fc78f5cb6-49w9v" event={"ID":"eebbc4b3-1248-4ef1-8c34-e2ca6008e746","Type":"ContainerDied","Data":"e246eae5f8fb1fb74989d74a43e2b0efdd97da236ac69204b198ce15ab81ae68"}
Sep 17 23:32:04.311042 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:32:04.310989    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-fc78f5cb6-49w9v" event={"ID":"eebbc4b3-1248-4ef1-8c34-e2ca6008e746","Type":"ContainerStarted","Data":"8b7bdefb61ce05ead665110f12d67aa785c12ecb9624589a38abe35f4ab426c2"}
Sep 17 23:32:04.311417 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:32:04.311068    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-fc78f5cb6-49w9v" event={"ID":"eebbc4b3-1248-4ef1-8c34-e2ca6008e746","Type":"ContainerStarted","Data":"ac85466b689e1f950855079061336104a69298c9f39e9359a6d8852ea1a1071f"}
Sep 17 23:32:04.311417 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:32:04.311262    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-fc78f5cb6-49w9v"
Sep 17 23:32:04.330069 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:32:04.330019    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-fc78f5cb6-49w9v" podStartSLOduration=3.657867295 podStartE2EDuration="7.329977011s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.672109716s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:31:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:32:04.329759197 +0000 UTC m=+1307.695419114" watchObservedRunningTime="2026-09-17 23:32:04.329977011 +0000 UTC m=+1307.695636928"
Sep 17 23:32:05.315813 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:32:05.315782    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-fc78f5cb6-49w9v"
Sep 17 23:32:05.317063 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:32:05.317036    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-fc78f5cb6-49w9v" podUID="eebbc4b3-1248-4ef1-8c34-e2ca6008e746" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Sep 17 23:32:06.320501 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:32:06.320454    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-fc78f5cb6-49w9v" podUID="eebbc4b3-1248-4ef1-8c34-e2ca6008e746" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Sep 17 23:32:11.325668 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:32:11.325640    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-fc78f5cb6-49w9v"
Sep 17 23:32:11.326186 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:32:11.326156    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-fc78f5cb6-49w9v" podUID="eebbc4b3-1248-4ef1-8c34-e2ca6008e746" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Sep 17 23:32:21.326315 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:32:21.326273    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-fc78f5cb6-49w9v" podUID="eebbc4b3-1248-4ef1-8c34-e2ca6008e746" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Sep 17 23:32:31.326730 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:32:31.326643    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-fc78f5cb6-49w9v" podUID="eebbc4b3-1248-4ef1-8c34-e2ca6008e746" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Sep 17 23:32:41.327001 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:32:41.326954    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-fc78f5cb6-49w9v" podUID="eebbc4b3-1248-4ef1-8c34-e2ca6008e746" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Sep 17 23:32:51.326593 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:32:51.326549    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-fc78f5cb6-49w9v" podUID="eebbc4b3-1248-4ef1-8c34-e2ca6008e746" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Sep 17 23:33:01.327106 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:33:01.327060    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-fc78f5cb6-49w9v" podUID="eebbc4b3-1248-4ef1-8c34-e2ca6008e746" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Sep 17 23:33:11.326134 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:33:11.326085    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-fc78f5cb6-49w9v" podUID="eebbc4b3-1248-4ef1-8c34-e2ca6008e746" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Sep 17 23:33:12.681472 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:33:12.681436    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:33:16.763275 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:33:16.763245    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-fc78f5cb6-49w9v"
Sep 17 23:33:27.826949 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:33:27.826905    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-fc78f5cb6-49w9v"]
Sep 17 23:33:27.827417 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:33:27.827167    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-fc78f5cb6-49w9v" podUID="eebbc4b3-1248-4ef1-8c34-e2ca6008e746" containerName="kserve-container" containerID="cri-o://ac85466b689e1f950855079061336104a69298c9f39e9359a6d8852ea1a1071f" gracePeriod=30
Sep 17 23:33:27.827417 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:33:27.827236    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-fc78f5cb6-49w9v" podUID="eebbc4b3-1248-4ef1-8c34-e2ca6008e746" containerName="kube-rbac-proxy" containerID="cri-o://8b7bdefb61ce05ead665110f12d67aa785c12ecb9624589a38abe35f4ab426c2" gracePeriod=30
Sep 17 23:33:27.852809 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:33:27.852781    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:33:27.874101 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:33:27.874069    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8555cb9565-sxxms"]
Sep 17 23:33:27.877680 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:33:27.877663    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8555cb9565-sxxms"
Sep 17 23:33:27.880080 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:33:27.880058    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-v2-predictor-serving-cert\""
Sep 17 23:33:27.890994 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:33:27.890966    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8555cb9565-sxxms"]
Sep 17 23:33:27.966864 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:33:27.966830    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/264c6ea8-f779-4c83-90a2-69b7ab92598d-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-8555cb9565-sxxms\" (UID: \"264c6ea8-f779-4c83-90a2-69b7ab92598d\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8555cb9565-sxxms"
Sep 17 23:33:27.967069 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:33:27.966884    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vt8l6\" (UniqueName: \"kubernetes.io/projected/264c6ea8-f779-4c83-90a2-69b7ab92598d-kube-api-access-vt8l6\") pod \"isvc-canary-promote-v2-predictor-8555cb9565-sxxms\" (UID: \"264c6ea8-f779-4c83-90a2-69b7ab92598d\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8555cb9565-sxxms"
Sep 17 23:33:27.967069 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:33:27.966942    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/264c6ea8-f779-4c83-90a2-69b7ab92598d-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-8555cb9565-sxxms\" (UID: \"264c6ea8-f779-4c83-90a2-69b7ab92598d\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8555cb9565-sxxms"
Sep 17 23:33:27.967069 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:33:27.967020    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/264c6ea8-f779-4c83-90a2-69b7ab92598d-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-8555cb9565-sxxms\" (UID: \"264c6ea8-f779-4c83-90a2-69b7ab92598d\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8555cb9565-sxxms"
Sep 17 23:33:27.967232 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:33:27.967109    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/264c6ea8-f779-4c83-90a2-69b7ab92598d-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-8555cb9565-sxxms\" (UID: \"264c6ea8-f779-4c83-90a2-69b7ab92598d\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8555cb9565-sxxms"
Sep 17 23:33:28.068055 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:33:28.067992    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/264c6ea8-f779-4c83-90a2-69b7ab92598d-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-8555cb9565-sxxms\" (UID: \"264c6ea8-f779-4c83-90a2-69b7ab92598d\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8555cb9565-sxxms"
Sep 17 23:33:28.068266 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:33:28.068080    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/264c6ea8-f779-4c83-90a2-69b7ab92598d-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-8555cb9565-sxxms\" (UID: \"264c6ea8-f779-4c83-90a2-69b7ab92598d\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8555cb9565-sxxms"
Sep 17 23:33:28.068266 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:33:28.068137    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/264c6ea8-f779-4c83-90a2-69b7ab92598d-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-8555cb9565-sxxms\" (UID: \"264c6ea8-f779-4c83-90a2-69b7ab92598d\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8555cb9565-sxxms"
Sep 17 23:33:28.068266 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:33:28.068157    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/264c6ea8-f779-4c83-90a2-69b7ab92598d-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-8555cb9565-sxxms\" (UID: \"264c6ea8-f779-4c83-90a2-69b7ab92598d\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8555cb9565-sxxms"
Sep 17 23:33:28.068266 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:33:28.068181    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vt8l6\" (UniqueName: \"kubernetes.io/projected/264c6ea8-f779-4c83-90a2-69b7ab92598d-kube-api-access-vt8l6\") pod \"isvc-canary-promote-v2-predictor-8555cb9565-sxxms\" (UID: \"264c6ea8-f779-4c83-90a2-69b7ab92598d\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8555cb9565-sxxms"
Sep 17 23:33:28.068500 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:33:28.068389    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/264c6ea8-f779-4c83-90a2-69b7ab92598d-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-8555cb9565-sxxms\" (UID: \"264c6ea8-f779-4c83-90a2-69b7ab92598d\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8555cb9565-sxxms"
Sep 17 23:33:28.068759 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:33:28.068738    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/264c6ea8-f779-4c83-90a2-69b7ab92598d-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-8555cb9565-sxxms\" (UID: \"264c6ea8-f779-4c83-90a2-69b7ab92598d\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8555cb9565-sxxms"
Sep 17 23:33:28.068854 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:33:28.068835    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/264c6ea8-f779-4c83-90a2-69b7ab92598d-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-8555cb9565-sxxms\" (UID: \"264c6ea8-f779-4c83-90a2-69b7ab92598d\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8555cb9565-sxxms"
Sep 17 23:33:28.070740 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:33:28.070705    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/264c6ea8-f779-4c83-90a2-69b7ab92598d-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-8555cb9565-sxxms\" (UID: \"264c6ea8-f779-4c83-90a2-69b7ab92598d\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8555cb9565-sxxms"
Sep 17 23:33:28.077773 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:33:28.077716    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vt8l6\" (UniqueName: \"kubernetes.io/projected/264c6ea8-f779-4c83-90a2-69b7ab92598d-kube-api-access-vt8l6\") pod \"isvc-canary-promote-v2-predictor-8555cb9565-sxxms\" (UID: \"264c6ea8-f779-4c83-90a2-69b7ab92598d\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8555cb9565-sxxms"
Sep 17 23:33:28.187969 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:33:28.187921    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8555cb9565-sxxms"
Sep 17 23:33:28.476524 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:33:28.476436    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8555cb9565-sxxms"]
Sep 17 23:33:28.479288 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:33:28.479267    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod264c6ea8_f779_4c83_90a2_69b7ab92598d.slice/crio-25fbc9e3b97b79b2d98775267907747bb3d427a1d516f99e507354adda940770 WatchSource:0}: Error finding container 25fbc9e3b97b79b2d98775267907747bb3d427a1d516f99e507354adda940770: Status 404 returned error can't find the container with id 25fbc9e3b97b79b2d98775267907747bb3d427a1d516f99e507354adda940770
Sep 17 23:33:28.660688 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:33:28.660648    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8555cb9565-sxxms" event={"ID":"264c6ea8-f779-4c83-90a2-69b7ab92598d","Type":"ContainerStarted","Data":"18e795c15c8888a847795b655a7ed7a858845ae0ee9ffc8bd57df5c3e807388f"}
Sep 17 23:33:28.660897 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:33:28.660718    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8555cb9565-sxxms" event={"ID":"264c6ea8-f779-4c83-90a2-69b7ab92598d","Type":"ContainerStarted","Data":"25fbc9e3b97b79b2d98775267907747bb3d427a1d516f99e507354adda940770"}
Sep 17 23:33:28.663195 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:33:28.663168    2584 generic.go:356] "Generic (PLEG): container finished" podID="eebbc4b3-1248-4ef1-8c34-e2ca6008e746" containerID="8b7bdefb61ce05ead665110f12d67aa785c12ecb9624589a38abe35f4ab426c2" exitCode=2
Sep 17 23:33:28.663313 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:33:28.663203    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-fc78f5cb6-49w9v" event={"ID":"eebbc4b3-1248-4ef1-8c34-e2ca6008e746","Type":"ContainerDied","Data":"8b7bdefb61ce05ead665110f12d67aa785c12ecb9624589a38abe35f4ab426c2"}
Sep 17 23:33:31.321092 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:33:31.321039    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-fc78f5cb6-49w9v" podUID="eebbc4b3-1248-4ef1-8c34-e2ca6008e746" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.29:8643/healthz\": dial tcp 10.132.0.29:8643: connect: connection refused"
Sep 17 23:33:32.679331 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:33:32.679257    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-fc78f5cb6-49w9v"
Sep 17 23:33:32.680114 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:33:32.680091    2584 generic.go:356] "Generic (PLEG): container finished" podID="264c6ea8-f779-4c83-90a2-69b7ab92598d" containerID="18e795c15c8888a847795b655a7ed7a858845ae0ee9ffc8bd57df5c3e807388f" exitCode=0
Sep 17 23:33:32.680211 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:33:32.680165    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8555cb9565-sxxms" event={"ID":"264c6ea8-f779-4c83-90a2-69b7ab92598d","Type":"ContainerDied","Data":"18e795c15c8888a847795b655a7ed7a858845ae0ee9ffc8bd57df5c3e807388f"}
Sep 17 23:33:32.682145 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:33:32.682123    2584 generic.go:356] "Generic (PLEG): container finished" podID="eebbc4b3-1248-4ef1-8c34-e2ca6008e746" containerID="ac85466b689e1f950855079061336104a69298c9f39e9359a6d8852ea1a1071f" exitCode=0
Sep 17 23:33:32.682233 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:33:32.682175    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-fc78f5cb6-49w9v" event={"ID":"eebbc4b3-1248-4ef1-8c34-e2ca6008e746","Type":"ContainerDied","Data":"ac85466b689e1f950855079061336104a69298c9f39e9359a6d8852ea1a1071f"}
Sep 17 23:33:32.682233 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:33:32.682185    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-fc78f5cb6-49w9v"
Sep 17 23:33:32.682233 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:33:32.682220    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-fc78f5cb6-49w9v" event={"ID":"eebbc4b3-1248-4ef1-8c34-e2ca6008e746","Type":"ContainerDied","Data":"f49d7e4fc4a75c0963a7f714bab9221f498a6a3cd58c6c49d355e2907a574fa7"}
Sep 17 23:33:32.682409 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:33:32.682247    2584 scope.go:117] "RemoveContainer" containerID="8b7bdefb61ce05ead665110f12d67aa785c12ecb9624589a38abe35f4ab426c2"
Sep 17 23:33:32.693785 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:33:32.693763    2584 scope.go:117] "RemoveContainer" containerID="ac85466b689e1f950855079061336104a69298c9f39e9359a6d8852ea1a1071f"
Sep 17 23:33:32.708616 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:33:32.708575    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ksv4d\" (UniqueName: \"kubernetes.io/projected/eebbc4b3-1248-4ef1-8c34-e2ca6008e746-kube-api-access-ksv4d\") pod \"eebbc4b3-1248-4ef1-8c34-e2ca6008e746\" (UID: \"eebbc4b3-1248-4ef1-8c34-e2ca6008e746\") "
Sep 17 23:33:32.708780 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:33:32.708629    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/eebbc4b3-1248-4ef1-8c34-e2ca6008e746-kserve-provision-location\") pod \"eebbc4b3-1248-4ef1-8c34-e2ca6008e746\" (UID: \"eebbc4b3-1248-4ef1-8c34-e2ca6008e746\") "
Sep 17 23:33:32.708780 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:33:32.708664    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/eebbc4b3-1248-4ef1-8c34-e2ca6008e746-cabundle-cert\") pod \"eebbc4b3-1248-4ef1-8c34-e2ca6008e746\" (UID: \"eebbc4b3-1248-4ef1-8c34-e2ca6008e746\") "
Sep 17 23:33:32.708780 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:33:32.708715    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/eebbc4b3-1248-4ef1-8c34-e2ca6008e746-proxy-tls\") pod \"eebbc4b3-1248-4ef1-8c34-e2ca6008e746\" (UID: \"eebbc4b3-1248-4ef1-8c34-e2ca6008e746\") "
Sep 17 23:33:32.708942 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:33:32.708791    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/eebbc4b3-1248-4ef1-8c34-e2ca6008e746-kube-rbac-proxy-sar-config\") pod \"eebbc4b3-1248-4ef1-8c34-e2ca6008e746\" (UID: \"eebbc4b3-1248-4ef1-8c34-e2ca6008e746\") "
Sep 17 23:33:32.709816 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:33:32.709095    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eebbc4b3-1248-4ef1-8c34-e2ca6008e746-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "eebbc4b3-1248-4ef1-8c34-e2ca6008e746" (UID: "eebbc4b3-1248-4ef1-8c34-e2ca6008e746"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:33:32.709816 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:33:32.709198    2584 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/eebbc4b3-1248-4ef1-8c34-e2ca6008e746-cabundle-cert\") on node \"ip-10-0-142-4.ec2.internal\" DevicePath \"\""
Sep 17 23:33:32.709816 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:33:32.709625    2584 scope.go:117] "RemoveContainer" containerID="e246eae5f8fb1fb74989d74a43e2b0efdd97da236ac69204b198ce15ab81ae68"
Sep 17 23:33:32.709816 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:33:32.709697    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eebbc4b3-1248-4ef1-8c34-e2ca6008e746-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "eebbc4b3-1248-4ef1-8c34-e2ca6008e746" (UID: "eebbc4b3-1248-4ef1-8c34-e2ca6008e746"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:33:32.710144 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:33:32.709842    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eebbc4b3-1248-4ef1-8c34-e2ca6008e746-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "eebbc4b3-1248-4ef1-8c34-e2ca6008e746" (UID: "eebbc4b3-1248-4ef1-8c34-e2ca6008e746"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:33:32.711138 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:33:32.711108    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eebbc4b3-1248-4ef1-8c34-e2ca6008e746-kube-api-access-ksv4d" (OuterVolumeSpecName: "kube-api-access-ksv4d") pod "eebbc4b3-1248-4ef1-8c34-e2ca6008e746" (UID: "eebbc4b3-1248-4ef1-8c34-e2ca6008e746"). InnerVolumeSpecName "kube-api-access-ksv4d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:33:32.711690 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:33:32.711668    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eebbc4b3-1248-4ef1-8c34-e2ca6008e746-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "eebbc4b3-1248-4ef1-8c34-e2ca6008e746" (UID: "eebbc4b3-1248-4ef1-8c34-e2ca6008e746"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:33:32.725334 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:33:32.725297    2584 scope.go:117] "RemoveContainer" containerID="8b7bdefb61ce05ead665110f12d67aa785c12ecb9624589a38abe35f4ab426c2"
Sep 17 23:33:32.725716 ip-10-0-142-4 kubenswrapper[2584]: E0917 23:33:32.725691    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b7bdefb61ce05ead665110f12d67aa785c12ecb9624589a38abe35f4ab426c2\": container with ID starting with 8b7bdefb61ce05ead665110f12d67aa785c12ecb9624589a38abe35f4ab426c2 not found: ID does not exist" containerID="8b7bdefb61ce05ead665110f12d67aa785c12ecb9624589a38abe35f4ab426c2"
Sep 17 23:33:32.725824 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:33:32.725725    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b7bdefb61ce05ead665110f12d67aa785c12ecb9624589a38abe35f4ab426c2"} err="failed to get container status \"8b7bdefb61ce05ead665110f12d67aa785c12ecb9624589a38abe35f4ab426c2\": rpc error: code = NotFound desc = could not find container \"8b7bdefb61ce05ead665110f12d67aa785c12ecb9624589a38abe35f4ab426c2\": container with ID starting with 8b7bdefb61ce05ead665110f12d67aa785c12ecb9624589a38abe35f4ab426c2 not found: ID does not exist"
Sep 17 23:33:32.725824 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:33:32.725744    2584 scope.go:117] "RemoveContainer" containerID="ac85466b689e1f950855079061336104a69298c9f39e9359a6d8852ea1a1071f"
Sep 17 23:33:32.726068 ip-10-0-142-4 kubenswrapper[2584]: E0917 23:33:32.726046    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac85466b689e1f950855079061336104a69298c9f39e9359a6d8852ea1a1071f\": container with ID starting with ac85466b689e1f950855079061336104a69298c9f39e9359a6d8852ea1a1071f not found: ID does not exist" containerID="ac85466b689e1f950855079061336104a69298c9f39e9359a6d8852ea1a1071f"
Sep 17 23:33:32.726164 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:33:32.726073    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac85466b689e1f950855079061336104a69298c9f39e9359a6d8852ea1a1071f"} err="failed to get container status \"ac85466b689e1f950855079061336104a69298c9f39e9359a6d8852ea1a1071f\": rpc error: code = NotFound desc = could not find container \"ac85466b689e1f950855079061336104a69298c9f39e9359a6d8852ea1a1071f\": container with ID starting with ac85466b689e1f950855079061336104a69298c9f39e9359a6d8852ea1a1071f not found: ID does not exist"
Sep 17 23:33:32.726164 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:33:32.726096    2584 scope.go:117] "RemoveContainer" containerID="e246eae5f8fb1fb74989d74a43e2b0efdd97da236ac69204b198ce15ab81ae68"
Sep 17 23:33:32.726400 ip-10-0-142-4 kubenswrapper[2584]: E0917 23:33:32.726379    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e246eae5f8fb1fb74989d74a43e2b0efdd97da236ac69204b198ce15ab81ae68\": container with ID starting with e246eae5f8fb1fb74989d74a43e2b0efdd97da236ac69204b198ce15ab81ae68 not found: ID does not exist" containerID="e246eae5f8fb1fb74989d74a43e2b0efdd97da236ac69204b198ce15ab81ae68"
Sep 17 23:33:32.726495 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:33:32.726403    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e246eae5f8fb1fb74989d74a43e2b0efdd97da236ac69204b198ce15ab81ae68"} err="failed to get container status \"e246eae5f8fb1fb74989d74a43e2b0efdd97da236ac69204b198ce15ab81ae68\": rpc error: code = NotFound desc = could not find container \"e246eae5f8fb1fb74989d74a43e2b0efdd97da236ac69204b198ce15ab81ae68\": container with ID starting with e246eae5f8fb1fb74989d74a43e2b0efdd97da236ac69204b198ce15ab81ae68 not found: ID does not exist"
Sep 17 23:33:32.809694 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:33:32.809666    2584 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/eebbc4b3-1248-4ef1-8c34-e2ca6008e746-proxy-tls\") on node \"ip-10-0-142-4.ec2.internal\" DevicePath \"\""
Sep 17 23:33:32.809787 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:33:32.809700    2584 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/eebbc4b3-1248-4ef1-8c34-e2ca6008e746-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-4.ec2.internal\" DevicePath \"\""
Sep 17 23:33:32.809787 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:33:32.809717    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ksv4d\" (UniqueName: \"kubernetes.io/projected/eebbc4b3-1248-4ef1-8c34-e2ca6008e746-kube-api-access-ksv4d\") on node \"ip-10-0-142-4.ec2.internal\" DevicePath \"\""
Sep 17 23:33:32.809787 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:33:32.809734    2584 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/eebbc4b3-1248-4ef1-8c34-e2ca6008e746-kserve-provision-location\") on node \"ip-10-0-142-4.ec2.internal\" DevicePath \"\""
Sep 17 23:33:32.997887 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:33:32.997788    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-fc78f5cb6-49w9v"]
Sep 17 23:33:33.002352 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:33:33.002320    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-fc78f5cb6-49w9v"]
Sep 17 23:33:33.688987 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:33:33.688905    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8555cb9565-sxxms" event={"ID":"264c6ea8-f779-4c83-90a2-69b7ab92598d","Type":"ContainerStarted","Data":"99b2ceda3dbb9f17b317cb0df85f46cdfe22e4f3d62b370a4245f648b9beb743"}
Sep 17 23:33:33.688987 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:33:33.688963    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8555cb9565-sxxms" event={"ID":"264c6ea8-f779-4c83-90a2-69b7ab92598d","Type":"ContainerStarted","Data":"8103856b1f6daf575d27de9d7880d40c389ba0cece63da3d97ff6aee3f9be1bb"}
Sep 17 23:33:33.689493 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:33:33.689318    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8555cb9565-sxxms"
Sep 17 23:33:33.689493 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:33:33.689461    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8555cb9565-sxxms"
Sep 17 23:33:33.690678 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:33:33.690654    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8555cb9565-sxxms" podUID="264c6ea8-f779-4c83-90a2-69b7ab92598d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Sep 17 23:33:33.707944 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:33:33.707890    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8555cb9565-sxxms" podStartSLOduration=2.827921882 podStartE2EDuration="6.707874503s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.879952621s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:33:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:33:33.706789263 +0000 UTC m=+1397.072449178" watchObservedRunningTime="2026-09-17 23:33:33.707874503 +0000 UTC m=+1397.073534493"
Sep 17 23:33:34.694232 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:33:34.694187    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8555cb9565-sxxms" podUID="264c6ea8-f779-4c83-90a2-69b7ab92598d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Sep 17 23:33:34.766237 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:33:34.766198    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eebbc4b3-1248-4ef1-8c34-e2ca6008e746" path="/var/lib/kubelet/pods/eebbc4b3-1248-4ef1-8c34-e2ca6008e746/volumes"
Sep 17 23:33:35.376362 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:33:35.376329    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:33:39.698715 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:33:39.698684    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8555cb9565-sxxms"
Sep 17 23:33:39.699252 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:33:39.699224    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8555cb9565-sxxms" podUID="264c6ea8-f779-4c83-90a2-69b7ab92598d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Sep 17 23:33:49.699467 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:33:49.699430    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8555cb9565-sxxms" podUID="264c6ea8-f779-4c83-90a2-69b7ab92598d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Sep 17 23:33:59.699638 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:33:59.699555    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8555cb9565-sxxms" podUID="264c6ea8-f779-4c83-90a2-69b7ab92598d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Sep 17 23:34:09.699156 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:34:09.699102    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8555cb9565-sxxms" podUID="264c6ea8-f779-4c83-90a2-69b7ab92598d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Sep 17 23:34:19.699705 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:34:19.699660    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8555cb9565-sxxms" podUID="264c6ea8-f779-4c83-90a2-69b7ab92598d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Sep 17 23:34:29.699597 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:34:29.699542    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8555cb9565-sxxms" podUID="264c6ea8-f779-4c83-90a2-69b7ab92598d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Sep 17 23:34:39.699259 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:34:39.699216    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8555cb9565-sxxms" podUID="264c6ea8-f779-4c83-90a2-69b7ab92598d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Sep 17 23:34:41.758159 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:34:41.758108    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8555cb9565-sxxms" podUID="264c6ea8-f779-4c83-90a2-69b7ab92598d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Sep 17 23:34:51.758703 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:34:51.758672    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8555cb9565-sxxms"
Sep 17 23:34:58.085273 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:34:58.085241    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8555cb9565-sxxms"]
Sep 17 23:34:58.085688 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:34:58.085520    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8555cb9565-sxxms" podUID="264c6ea8-f779-4c83-90a2-69b7ab92598d" containerName="kserve-container" containerID="cri-o://8103856b1f6daf575d27de9d7880d40c389ba0cece63da3d97ff6aee3f9be1bb" gracePeriod=30
Sep 17 23:34:58.085688 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:34:58.085585    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8555cb9565-sxxms" podUID="264c6ea8-f779-4c83-90a2-69b7ab92598d" containerName="kube-rbac-proxy" containerID="cri-o://99b2ceda3dbb9f17b317cb0df85f46cdfe22e4f3d62b370a4245f648b9beb743" gracePeriod=30
Sep 17 23:34:58.115671 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:34:58.115645    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:34:59.037097 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:34:59.037063    2584 generic.go:356] "Generic (PLEG): container finished" podID="264c6ea8-f779-4c83-90a2-69b7ab92598d" containerID="99b2ceda3dbb9f17b317cb0df85f46cdfe22e4f3d62b370a4245f648b9beb743" exitCode=2
Sep 17 23:34:59.037264 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:34:59.037136    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8555cb9565-sxxms" event={"ID":"264c6ea8-f779-4c83-90a2-69b7ab92598d","Type":"ContainerDied","Data":"99b2ceda3dbb9f17b317cb0df85f46cdfe22e4f3d62b370a4245f648b9beb743"}
Sep 17 23:34:59.694881 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:34:59.694836    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8555cb9565-sxxms" podUID="264c6ea8-f779-4c83-90a2-69b7ab92598d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.30:8643/healthz\": dial tcp 10.132.0.30:8643: connect: connection refused"
Sep 17 23:35:01.758890 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:35:01.758846    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8555cb9565-sxxms" podUID="264c6ea8-f779-4c83-90a2-69b7ab92598d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Sep 17 23:35:03.029257 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:35:03.029231    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8555cb9565-sxxms"
Sep 17 23:35:03.060184 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:35:03.060148    2584 generic.go:356] "Generic (PLEG): container finished" podID="264c6ea8-f779-4c83-90a2-69b7ab92598d" containerID="8103856b1f6daf575d27de9d7880d40c389ba0cece63da3d97ff6aee3f9be1bb" exitCode=0
Sep 17 23:35:03.060366 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:35:03.060193    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8555cb9565-sxxms" event={"ID":"264c6ea8-f779-4c83-90a2-69b7ab92598d","Type":"ContainerDied","Data":"8103856b1f6daf575d27de9d7880d40c389ba0cece63da3d97ff6aee3f9be1bb"}
Sep 17 23:35:03.060366 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:35:03.060207    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8555cb9565-sxxms"
Sep 17 23:35:03.060366 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:35:03.060244    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8555cb9565-sxxms" event={"ID":"264c6ea8-f779-4c83-90a2-69b7ab92598d","Type":"ContainerDied","Data":"25fbc9e3b97b79b2d98775267907747bb3d427a1d516f99e507354adda940770"}
Sep 17 23:35:03.060366 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:35:03.060271    2584 scope.go:117] "RemoveContainer" containerID="99b2ceda3dbb9f17b317cb0df85f46cdfe22e4f3d62b370a4245f648b9beb743"
Sep 17 23:35:03.068937 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:35:03.068914    2584 scope.go:117] "RemoveContainer" containerID="8103856b1f6daf575d27de9d7880d40c389ba0cece63da3d97ff6aee3f9be1bb"
Sep 17 23:35:03.076806 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:35:03.076787    2584 scope.go:117] "RemoveContainer" containerID="18e795c15c8888a847795b655a7ed7a858845ae0ee9ffc8bd57df5c3e807388f"
Sep 17 23:35:03.084202 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:35:03.084183    2584 scope.go:117] "RemoveContainer" containerID="99b2ceda3dbb9f17b317cb0df85f46cdfe22e4f3d62b370a4245f648b9beb743"
Sep 17 23:35:03.084480 ip-10-0-142-4 kubenswrapper[2584]: E0917 23:35:03.084464    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"99b2ceda3dbb9f17b317cb0df85f46cdfe22e4f3d62b370a4245f648b9beb743\": container with ID starting with 99b2ceda3dbb9f17b317cb0df85f46cdfe22e4f3d62b370a4245f648b9beb743 not found: ID does not exist" containerID="99b2ceda3dbb9f17b317cb0df85f46cdfe22e4f3d62b370a4245f648b9beb743"
Sep 17 23:35:03.084540 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:35:03.084489    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"99b2ceda3dbb9f17b317cb0df85f46cdfe22e4f3d62b370a4245f648b9beb743"} err="failed to get container status \"99b2ceda3dbb9f17b317cb0df85f46cdfe22e4f3d62b370a4245f648b9beb743\": rpc error: code = NotFound desc = could not find container \"99b2ceda3dbb9f17b317cb0df85f46cdfe22e4f3d62b370a4245f648b9beb743\": container with ID starting with 99b2ceda3dbb9f17b317cb0df85f46cdfe22e4f3d62b370a4245f648b9beb743 not found: ID does not exist"
Sep 17 23:35:03.084540 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:35:03.084505    2584 scope.go:117] "RemoveContainer" containerID="8103856b1f6daf575d27de9d7880d40c389ba0cece63da3d97ff6aee3f9be1bb"
Sep 17 23:35:03.084738 ip-10-0-142-4 kubenswrapper[2584]: E0917 23:35:03.084722    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8103856b1f6daf575d27de9d7880d40c389ba0cece63da3d97ff6aee3f9be1bb\": container with ID starting with 8103856b1f6daf575d27de9d7880d40c389ba0cece63da3d97ff6aee3f9be1bb not found: ID does not exist" containerID="8103856b1f6daf575d27de9d7880d40c389ba0cece63da3d97ff6aee3f9be1bb"
Sep 17 23:35:03.084783 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:35:03.084745    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8103856b1f6daf575d27de9d7880d40c389ba0cece63da3d97ff6aee3f9be1bb"} err="failed to get container status \"8103856b1f6daf575d27de9d7880d40c389ba0cece63da3d97ff6aee3f9be1bb\": rpc error: code = NotFound desc = could not find container \"8103856b1f6daf575d27de9d7880d40c389ba0cece63da3d97ff6aee3f9be1bb\": container with ID starting with 8103856b1f6daf575d27de9d7880d40c389ba0cece63da3d97ff6aee3f9be1bb not found: ID does not exist"
Sep 17 23:35:03.084783 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:35:03.084759    2584 scope.go:117] "RemoveContainer" containerID="18e795c15c8888a847795b655a7ed7a858845ae0ee9ffc8bd57df5c3e807388f"
Sep 17 23:35:03.084958 ip-10-0-142-4 kubenswrapper[2584]: E0917 23:35:03.084944    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"18e795c15c8888a847795b655a7ed7a858845ae0ee9ffc8bd57df5c3e807388f\": container with ID starting with 18e795c15c8888a847795b655a7ed7a858845ae0ee9ffc8bd57df5c3e807388f not found: ID does not exist" containerID="18e795c15c8888a847795b655a7ed7a858845ae0ee9ffc8bd57df5c3e807388f"
Sep 17 23:35:03.085028 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:35:03.084961    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18e795c15c8888a847795b655a7ed7a858845ae0ee9ffc8bd57df5c3e807388f"} err="failed to get container status \"18e795c15c8888a847795b655a7ed7a858845ae0ee9ffc8bd57df5c3e807388f\": rpc error: code = NotFound desc = could not find container \"18e795c15c8888a847795b655a7ed7a858845ae0ee9ffc8bd57df5c3e807388f\": container with ID starting with 18e795c15c8888a847795b655a7ed7a858845ae0ee9ffc8bd57df5c3e807388f not found: ID does not exist"
Sep 17 23:35:03.182566 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:35:03.182525    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/264c6ea8-f779-4c83-90a2-69b7ab92598d-kube-rbac-proxy-sar-config\") pod \"264c6ea8-f779-4c83-90a2-69b7ab92598d\" (UID: \"264c6ea8-f779-4c83-90a2-69b7ab92598d\") "
Sep 17 23:35:03.182763 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:35:03.182592    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vt8l6\" (UniqueName: \"kubernetes.io/projected/264c6ea8-f779-4c83-90a2-69b7ab92598d-kube-api-access-vt8l6\") pod \"264c6ea8-f779-4c83-90a2-69b7ab92598d\" (UID: \"264c6ea8-f779-4c83-90a2-69b7ab92598d\") "
Sep 17 23:35:03.182763 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:35:03.182620    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/264c6ea8-f779-4c83-90a2-69b7ab92598d-kserve-provision-location\") pod \"264c6ea8-f779-4c83-90a2-69b7ab92598d\" (UID: \"264c6ea8-f779-4c83-90a2-69b7ab92598d\") "
Sep 17 23:35:03.182763 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:35:03.182668    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/264c6ea8-f779-4c83-90a2-69b7ab92598d-proxy-tls\") pod \"264c6ea8-f779-4c83-90a2-69b7ab92598d\" (UID: \"264c6ea8-f779-4c83-90a2-69b7ab92598d\") "
Sep 17 23:35:03.182763 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:35:03.182712    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/264c6ea8-f779-4c83-90a2-69b7ab92598d-cabundle-cert\") pod \"264c6ea8-f779-4c83-90a2-69b7ab92598d\" (UID: \"264c6ea8-f779-4c83-90a2-69b7ab92598d\") "
Sep 17 23:35:03.183107 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:35:03.183078    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/264c6ea8-f779-4c83-90a2-69b7ab92598d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "264c6ea8-f779-4c83-90a2-69b7ab92598d" (UID: "264c6ea8-f779-4c83-90a2-69b7ab92598d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:35:03.183204 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:35:03.183097    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/264c6ea8-f779-4c83-90a2-69b7ab92598d-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "264c6ea8-f779-4c83-90a2-69b7ab92598d" (UID: "264c6ea8-f779-4c83-90a2-69b7ab92598d"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:35:03.183204 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:35:03.183154    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/264c6ea8-f779-4c83-90a2-69b7ab92598d-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "264c6ea8-f779-4c83-90a2-69b7ab92598d" (UID: "264c6ea8-f779-4c83-90a2-69b7ab92598d"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:35:03.184998 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:35:03.184967    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/264c6ea8-f779-4c83-90a2-69b7ab92598d-kube-api-access-vt8l6" (OuterVolumeSpecName: "kube-api-access-vt8l6") pod "264c6ea8-f779-4c83-90a2-69b7ab92598d" (UID: "264c6ea8-f779-4c83-90a2-69b7ab92598d"). InnerVolumeSpecName "kube-api-access-vt8l6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:35:03.185132 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:35:03.185023    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/264c6ea8-f779-4c83-90a2-69b7ab92598d-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "264c6ea8-f779-4c83-90a2-69b7ab92598d" (UID: "264c6ea8-f779-4c83-90a2-69b7ab92598d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:35:03.283465 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:35:03.283423    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vt8l6\" (UniqueName: \"kubernetes.io/projected/264c6ea8-f779-4c83-90a2-69b7ab92598d-kube-api-access-vt8l6\") on node \"ip-10-0-142-4.ec2.internal\" DevicePath \"\""
Sep 17 23:35:03.283465 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:35:03.283458    2584 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/264c6ea8-f779-4c83-90a2-69b7ab92598d-kserve-provision-location\") on node \"ip-10-0-142-4.ec2.internal\" DevicePath \"\""
Sep 17 23:35:03.283465 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:35:03.283471    2584 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/264c6ea8-f779-4c83-90a2-69b7ab92598d-proxy-tls\") on node \"ip-10-0-142-4.ec2.internal\" DevicePath \"\""
Sep 17 23:35:03.283708 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:35:03.283479    2584 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/264c6ea8-f779-4c83-90a2-69b7ab92598d-cabundle-cert\") on node \"ip-10-0-142-4.ec2.internal\" DevicePath \"\""
Sep 17 23:35:03.283708 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:35:03.283490    2584 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/264c6ea8-f779-4c83-90a2-69b7ab92598d-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-4.ec2.internal\" DevicePath \"\""
Sep 17 23:35:03.390020 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:35:03.389973    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8555cb9565-sxxms"]
Sep 17 23:35:03.393745 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:35:03.393718    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8555cb9565-sxxms"]
Sep 17 23:35:04.762597 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:35:04.762564    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="264c6ea8-f779-4c83-90a2-69b7ab92598d" path="/var/lib/kubelet/pods/264c6ea8-f779-4c83-90a2-69b7ab92598d/volumes"
Sep 17 23:35:12.170968 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:35:12.170932    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:35:12.214143 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:35:12.214102    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-6d2jt"]
Sep 17 23:35:12.214449 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:35:12.214437    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eebbc4b3-1248-4ef1-8c34-e2ca6008e746" containerName="kserve-container"
Sep 17 23:35:12.214493 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:35:12.214453    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="eebbc4b3-1248-4ef1-8c34-e2ca6008e746" containerName="kserve-container"
Sep 17 23:35:12.214493 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:35:12.214464    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="264c6ea8-f779-4c83-90a2-69b7ab92598d" containerName="storage-initializer"
Sep 17 23:35:12.214493 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:35:12.214470    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="264c6ea8-f779-4c83-90a2-69b7ab92598d" containerName="storage-initializer"
Sep 17 23:35:12.214493 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:35:12.214484    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="264c6ea8-f779-4c83-90a2-69b7ab92598d" containerName="kserve-container"
Sep 17 23:35:12.214493 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:35:12.214492    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="264c6ea8-f779-4c83-90a2-69b7ab92598d" containerName="kserve-container"
Sep 17 23:35:12.214654 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:35:12.214499    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eebbc4b3-1248-4ef1-8c34-e2ca6008e746" containerName="kube-rbac-proxy"
Sep 17 23:35:12.214654 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:35:12.214505    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="eebbc4b3-1248-4ef1-8c34-e2ca6008e746" containerName="kube-rbac-proxy"
Sep 17 23:35:12.214654 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:35:12.214511    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="264c6ea8-f779-4c83-90a2-69b7ab92598d" containerName="kube-rbac-proxy"
Sep 17 23:35:12.214654 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:35:12.214516    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="264c6ea8-f779-4c83-90a2-69b7ab92598d" containerName="kube-rbac-proxy"
Sep 17 23:35:12.214654 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:35:12.214524    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eebbc4b3-1248-4ef1-8c34-e2ca6008e746" containerName="storage-initializer"
Sep 17 23:35:12.214654 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:35:12.214532    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="eebbc4b3-1248-4ef1-8c34-e2ca6008e746" containerName="storage-initializer"
Sep 17 23:35:12.214654 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:35:12.214579    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="264c6ea8-f779-4c83-90a2-69b7ab92598d" containerName="kserve-container"
Sep 17 23:35:12.214654 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:35:12.214585    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="eebbc4b3-1248-4ef1-8c34-e2ca6008e746" containerName="kserve-container"
Sep 17 23:35:12.214654 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:35:12.214593    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="264c6ea8-f779-4c83-90a2-69b7ab92598d" containerName="kube-rbac-proxy"
Sep 17 23:35:12.214654 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:35:12.214599    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="eebbc4b3-1248-4ef1-8c34-e2ca6008e746" containerName="kube-rbac-proxy"
Sep 17 23:35:12.217648 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:35:12.217628    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-6d2jt"
Sep 17 23:35:12.220175 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:35:12.220155    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-raw-cipn-predictor-serving-cert\""
Sep 17 23:35:12.220175 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:35:12.220171    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-raw-cipn-kube-rbac-proxy-sar-config\""
Sep 17 23:35:12.220352 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:35:12.220216    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 23:35:12.221199 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:35:12.221181    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-f2cbg\""
Sep 17 23:35:12.221296 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:35:12.221210    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 23:35:12.227978 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:35:12.227956    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-6d2jt"]
Sep 17 23:35:12.258404 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:35:12.258368    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-shmkb\" (UniqueName: \"kubernetes.io/projected/d8f4cba2-2fae-4416-aedf-c6ba28bb5385-kube-api-access-shmkb\") pod \"message-dumper-raw-cipn-predictor-9774bf55d-6d2jt\" (UID: \"d8f4cba2-2fae-4416-aedf-c6ba28bb5385\") " pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-6d2jt"
Sep 17 23:35:12.258561 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:35:12.258418    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d8f4cba2-2fae-4416-aedf-c6ba28bb5385-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-cipn-predictor-9774bf55d-6d2jt\" (UID: \"d8f4cba2-2fae-4416-aedf-c6ba28bb5385\") " pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-6d2jt"
Sep 17 23:35:12.258561 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:35:12.258447    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d8f4cba2-2fae-4416-aedf-c6ba28bb5385-proxy-tls\") pod \"message-dumper-raw-cipn-predictor-9774bf55d-6d2jt\" (UID: \"d8f4cba2-2fae-4416-aedf-c6ba28bb5385\") " pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-6d2jt"
Sep 17 23:35:12.359287 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:35:12.359252    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d8f4cba2-2fae-4416-aedf-c6ba28bb5385-proxy-tls\") pod \"message-dumper-raw-cipn-predictor-9774bf55d-6d2jt\" (UID: \"d8f4cba2-2fae-4416-aedf-c6ba28bb5385\") " pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-6d2jt"
Sep 17 23:35:12.359484 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:35:12.359319    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-shmkb\" (UniqueName: \"kubernetes.io/projected/d8f4cba2-2fae-4416-aedf-c6ba28bb5385-kube-api-access-shmkb\") pod \"message-dumper-raw-cipn-predictor-9774bf55d-6d2jt\" (UID: \"d8f4cba2-2fae-4416-aedf-c6ba28bb5385\") " pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-6d2jt"
Sep 17 23:35:12.359484 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:35:12.359360    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d8f4cba2-2fae-4416-aedf-c6ba28bb5385-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-cipn-predictor-9774bf55d-6d2jt\" (UID: \"d8f4cba2-2fae-4416-aedf-c6ba28bb5385\") " pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-6d2jt"
Sep 17 23:35:12.359484 ip-10-0-142-4 kubenswrapper[2584]: E0917 23:35:12.359419    2584 secret.go:189] Couldn't get secret kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-serving-cert: secret "message-dumper-raw-cipn-predictor-serving-cert" not found
Sep 17 23:35:12.359657 ip-10-0-142-4 kubenswrapper[2584]: E0917 23:35:12.359505    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d8f4cba2-2fae-4416-aedf-c6ba28bb5385-proxy-tls podName:d8f4cba2-2fae-4416-aedf-c6ba28bb5385 nodeName:}" failed. No retries permitted until 2026-09-17 23:35:12.859482938 +0000 UTC m=+1496.225142833 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/d8f4cba2-2fae-4416-aedf-c6ba28bb5385-proxy-tls") pod "message-dumper-raw-cipn-predictor-9774bf55d-6d2jt" (UID: "d8f4cba2-2fae-4416-aedf-c6ba28bb5385") : secret "message-dumper-raw-cipn-predictor-serving-cert" not found
Sep 17 23:35:12.359963 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:35:12.359943    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d8f4cba2-2fae-4416-aedf-c6ba28bb5385-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-cipn-predictor-9774bf55d-6d2jt\" (UID: \"d8f4cba2-2fae-4416-aedf-c6ba28bb5385\") " pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-6d2jt"
Sep 17 23:35:12.368020 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:35:12.367974    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-shmkb\" (UniqueName: \"kubernetes.io/projected/d8f4cba2-2fae-4416-aedf-c6ba28bb5385-kube-api-access-shmkb\") pod \"message-dumper-raw-cipn-predictor-9774bf55d-6d2jt\" (UID: \"d8f4cba2-2fae-4416-aedf-c6ba28bb5385\") " pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-6d2jt"
Sep 17 23:35:12.458547 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:35:12.458463    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:35:12.863850 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:35:12.863816    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d8f4cba2-2fae-4416-aedf-c6ba28bb5385-proxy-tls\") pod \"message-dumper-raw-cipn-predictor-9774bf55d-6d2jt\" (UID: \"d8f4cba2-2fae-4416-aedf-c6ba28bb5385\") " pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-6d2jt"
Sep 17 23:35:12.866440 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:35:12.866420    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d8f4cba2-2fae-4416-aedf-c6ba28bb5385-proxy-tls\") pod \"message-dumper-raw-cipn-predictor-9774bf55d-6d2jt\" (UID: \"d8f4cba2-2fae-4416-aedf-c6ba28bb5385\") " pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-6d2jt"
Sep 17 23:35:13.128731 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:35:13.128634    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-6d2jt"
Sep 17 23:35:13.407660 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:35:13.407628    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-6d2jt"]
Sep 17 23:35:13.409779 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:35:13.409749    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8f4cba2_2fae_4416_aedf_c6ba28bb5385.slice/crio-6d00e0df56e37f70b3f8a7f1e488bcedf5479847615d92723ad2072f05d6952c WatchSource:0}: Error finding container 6d00e0df56e37f70b3f8a7f1e488bcedf5479847615d92723ad2072f05d6952c: Status 404 returned error can't find the container with id 6d00e0df56e37f70b3f8a7f1e488bcedf5479847615d92723ad2072f05d6952c
Sep 17 23:35:13.411623 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:35:13.411607    2584 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 23:35:14.105778 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:35:14.105743    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-6d2jt" event={"ID":"d8f4cba2-2fae-4416-aedf-c6ba28bb5385","Type":"ContainerStarted","Data":"6d00e0df56e37f70b3f8a7f1e488bcedf5479847615d92723ad2072f05d6952c"}
Sep 17 23:35:15.111307 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:35:15.111273    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-6d2jt" event={"ID":"d8f4cba2-2fae-4416-aedf-c6ba28bb5385","Type":"ContainerStarted","Data":"f256bccea2348e4371fff5df8bf6a1ce9cd7c9c840ede619fc04d505028b3569"}
Sep 17 23:35:15.111752 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:35:15.111338    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-6d2jt" event={"ID":"d8f4cba2-2fae-4416-aedf-c6ba28bb5385","Type":"ContainerStarted","Data":"c3e7e8ac59770882a54c0bd6d2f3be782629af629ca6d091728d30253bea083f"}
Sep 17 23:35:15.111752 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:35:15.111484    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-6d2jt"
Sep 17 23:35:15.111752 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:35:15.111538    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-6d2jt"
Sep 17 23:35:15.113184 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:35:15.113161    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-6d2jt"
Sep 17 23:35:15.130149 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:35:15.130102    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-6d2jt" podStartSLOduration=1.8724953819999999 podStartE2EDuration="3.130087476s" totalImagesPullingTime="1.257592094s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:35:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:35:15.128914296 +0000 UTC m=+1498.494574214" watchObservedRunningTime="2026-09-17 23:35:15.130087476 +0000 UTC m=+1498.495747392"
Sep 17 23:35:22.126301 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:35:22.126272    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-6d2jt"
Sep 17 23:35:32.246519 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:35:32.246411    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:35:32.285794 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:35:32.285756    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-856f4c4b54-grmbw"]
Sep 17 23:35:32.289908 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:35:32.289888    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-856f4c4b54-grmbw"
Sep 17 23:35:32.292396 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:35:32.292377    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-raw-cipn-predictor-serving-cert\""
Sep 17 23:35:32.292785 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:35:32.292761    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-raw-cipn-kube-rbac-proxy-sar-config\""
Sep 17 23:35:32.292972 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:35:32.292954    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 23:35:32.293102 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:35:32.292986    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 23:35:32.303597 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:35:32.303572    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-856f4c4b54-grmbw"]
Sep 17 23:35:32.423691 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:35:32.423652    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rztnd\" (UniqueName: \"kubernetes.io/projected/9787d7a5-4cc3-4c33-a5d4-f869ffca714a-kube-api-access-rztnd\") pod \"isvc-logger-raw-cipn-predictor-856f4c4b54-grmbw\" (UID: \"9787d7a5-4cc3-4c33-a5d4-f869ffca714a\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-856f4c4b54-grmbw"
Sep 17 23:35:32.423691 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:35:32.423692    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9787d7a5-4cc3-4c33-a5d4-f869ffca714a-kserve-provision-location\") pod \"isvc-logger-raw-cipn-predictor-856f4c4b54-grmbw\" (UID: \"9787d7a5-4cc3-4c33-a5d4-f869ffca714a\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-856f4c4b54-grmbw"
Sep 17 23:35:32.423916 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:35:32.423712    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9787d7a5-4cc3-4c33-a5d4-f869ffca714a-cabundle-cert\") pod \"isvc-logger-raw-cipn-predictor-856f4c4b54-grmbw\" (UID: \"9787d7a5-4cc3-4c33-a5d4-f869ffca714a\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-856f4c4b54-grmbw"
Sep 17 23:35:32.423916 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:35:32.423806    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9787d7a5-4cc3-4c33-a5d4-f869ffca714a-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-cipn-predictor-856f4c4b54-grmbw\" (UID: \"9787d7a5-4cc3-4c33-a5d4-f869ffca714a\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-856f4c4b54-grmbw"
Sep 17 23:35:32.423916 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:35:32.423877    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9787d7a5-4cc3-4c33-a5d4-f869ffca714a-proxy-tls\") pod \"isvc-logger-raw-cipn-predictor-856f4c4b54-grmbw\" (UID: \"9787d7a5-4cc3-4c33-a5d4-f869ffca714a\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-856f4c4b54-grmbw"
Sep 17 23:35:32.516919 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:35:32.516887    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:35:32.525065 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:35:32.525044    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9787d7a5-4cc3-4c33-a5d4-f869ffca714a-cabundle-cert\") pod \"isvc-logger-raw-cipn-predictor-856f4c4b54-grmbw\" (UID: \"9787d7a5-4cc3-4c33-a5d4-f869ffca714a\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-856f4c4b54-grmbw"
Sep 17 23:35:32.525139 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:35:32.525087    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9787d7a5-4cc3-4c33-a5d4-f869ffca714a-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-cipn-predictor-856f4c4b54-grmbw\" (UID: \"9787d7a5-4cc3-4c33-a5d4-f869ffca714a\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-856f4c4b54-grmbw"
Sep 17 23:35:32.525139 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:35:32.525125    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9787d7a5-4cc3-4c33-a5d4-f869ffca714a-proxy-tls\") pod \"isvc-logger-raw-cipn-predictor-856f4c4b54-grmbw\" (UID: \"9787d7a5-4cc3-4c33-a5d4-f869ffca714a\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-856f4c4b54-grmbw"
Sep 17 23:35:32.525232 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:35:32.525164    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rztnd\" (UniqueName: \"kubernetes.io/projected/9787d7a5-4cc3-4c33-a5d4-f869ffca714a-kube-api-access-rztnd\") pod \"isvc-logger-raw-cipn-predictor-856f4c4b54-grmbw\" (UID: \"9787d7a5-4cc3-4c33-a5d4-f869ffca714a\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-856f4c4b54-grmbw"
Sep 17 23:35:32.525298 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:35:32.525266    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9787d7a5-4cc3-4c33-a5d4-f869ffca714a-kserve-provision-location\") pod \"isvc-logger-raw-cipn-predictor-856f4c4b54-grmbw\" (UID: \"9787d7a5-4cc3-4c33-a5d4-f869ffca714a\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-856f4c4b54-grmbw"
Sep 17 23:35:32.525642 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:35:32.525617    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9787d7a5-4cc3-4c33-a5d4-f869ffca714a-kserve-provision-location\") pod \"isvc-logger-raw-cipn-predictor-856f4c4b54-grmbw\" (UID: \"9787d7a5-4cc3-4c33-a5d4-f869ffca714a\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-856f4c4b54-grmbw"
Sep 17 23:35:32.525746 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:35:32.525728    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9787d7a5-4cc3-4c33-a5d4-f869ffca714a-cabundle-cert\") pod \"isvc-logger-raw-cipn-predictor-856f4c4b54-grmbw\" (UID: \"9787d7a5-4cc3-4c33-a5d4-f869ffca714a\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-856f4c4b54-grmbw"
Sep 17 23:35:32.525856 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:35:32.525837    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9787d7a5-4cc3-4c33-a5d4-f869ffca714a-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-cipn-predictor-856f4c4b54-grmbw\" (UID: \"9787d7a5-4cc3-4c33-a5d4-f869ffca714a\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-856f4c4b54-grmbw"
Sep 17 23:35:32.527791 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:35:32.527770    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9787d7a5-4cc3-4c33-a5d4-f869ffca714a-proxy-tls\") pod \"isvc-logger-raw-cipn-predictor-856f4c4b54-grmbw\" (UID: \"9787d7a5-4cc3-4c33-a5d4-f869ffca714a\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-856f4c4b54-grmbw"
Sep 17 23:35:32.534538 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:35:32.534514    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rztnd\" (UniqueName: \"kubernetes.io/projected/9787d7a5-4cc3-4c33-a5d4-f869ffca714a-kube-api-access-rztnd\") pod \"isvc-logger-raw-cipn-predictor-856f4c4b54-grmbw\" (UID: \"9787d7a5-4cc3-4c33-a5d4-f869ffca714a\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-856f4c4b54-grmbw"
Sep 17 23:35:32.599903 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:35:32.599867    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-856f4c4b54-grmbw"
Sep 17 23:35:32.877738 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:35:32.877710    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-856f4c4b54-grmbw"]
Sep 17 23:35:32.879763 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:35:32.879731    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9787d7a5_4cc3_4c33_a5d4_f869ffca714a.slice/crio-8b3aeaa88f7f683ebd68f47a26f82e8f7f255b410493478472e120931dd972f8 WatchSource:0}: Error finding container 8b3aeaa88f7f683ebd68f47a26f82e8f7f255b410493478472e120931dd972f8: Status 404 returned error can't find the container with id 8b3aeaa88f7f683ebd68f47a26f82e8f7f255b410493478472e120931dd972f8
Sep 17 23:35:33.186796 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:35:33.186705    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-856f4c4b54-grmbw" event={"ID":"9787d7a5-4cc3-4c33-a5d4-f869ffca714a","Type":"ContainerStarted","Data":"f49f84bbce952dd3b6fe41ee0d45677ede2b44afb75dfa4ef707afa4546be944"}
Sep 17 23:35:33.186947 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:35:33.186795    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-856f4c4b54-grmbw" event={"ID":"9787d7a5-4cc3-4c33-a5d4-f869ffca714a","Type":"ContainerStarted","Data":"8b3aeaa88f7f683ebd68f47a26f82e8f7f255b410493478472e120931dd972f8"}
Sep 17 23:35:37.204722 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:35:37.204629    2584 generic.go:356] "Generic (PLEG): container finished" podID="9787d7a5-4cc3-4c33-a5d4-f869ffca714a" containerID="f49f84bbce952dd3b6fe41ee0d45677ede2b44afb75dfa4ef707afa4546be944" exitCode=0
Sep 17 23:35:37.204722 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:35:37.204704    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-856f4c4b54-grmbw" event={"ID":"9787d7a5-4cc3-4c33-a5d4-f869ffca714a","Type":"ContainerDied","Data":"f49f84bbce952dd3b6fe41ee0d45677ede2b44afb75dfa4ef707afa4546be944"}
Sep 17 23:35:38.211639 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:35:38.211604    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-856f4c4b54-grmbw" event={"ID":"9787d7a5-4cc3-4c33-a5d4-f869ffca714a","Type":"ContainerStarted","Data":"4826cefc99c2b7fc404f8543ea7015c7794548e4804618c7272691f410f11851"}
Sep 17 23:35:38.212091 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:35:38.211673    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-856f4c4b54-grmbw" event={"ID":"9787d7a5-4cc3-4c33-a5d4-f869ffca714a","Type":"ContainerStarted","Data":"4a1cfd4ef041455a0258b7bdae9e4fd5a3ff10fdab94d8c166ab62510bd35d73"}
Sep 17 23:35:38.212091 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:35:38.211698    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-856f4c4b54-grmbw" event={"ID":"9787d7a5-4cc3-4c33-a5d4-f869ffca714a","Type":"ContainerStarted","Data":"726e8a6056a2475b892ad9f7c8cbf029801142fab4ec3fd6b8f5ff805c36a828"}
Sep 17 23:35:38.212091 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:35:38.211991    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-856f4c4b54-grmbw"
Sep 17 23:35:38.212091 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:35:38.212070    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-856f4c4b54-grmbw"
Sep 17 23:35:38.212091 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:35:38.212095    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-856f4c4b54-grmbw"
Sep 17 23:35:38.213531 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:35:38.213505    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-856f4c4b54-grmbw" podUID="9787d7a5-4cc3-4c33-a5d4-f869ffca714a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Sep 17 23:35:38.214236 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:35:38.214215    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-856f4c4b54-grmbw" podUID="9787d7a5-4cc3-4c33-a5d4-f869ffca714a" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 23:35:38.240900 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:35:38.240842    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-856f4c4b54-grmbw" podStartSLOduration=2.229923877 podStartE2EDuration="6.240823158s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.010899281s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:35:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:35:38.24010546 +0000 UTC m=+1521.605765377" watchObservedRunningTime="2026-09-17 23:35:38.240823158 +0000 UTC m=+1521.606483075"
Sep 17 23:35:39.216245 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:35:39.216206    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-856f4c4b54-grmbw" podUID="9787d7a5-4cc3-4c33-a5d4-f869ffca714a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Sep 17 23:35:39.216727 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:35:39.216699    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-856f4c4b54-grmbw" podUID="9787d7a5-4cc3-4c33-a5d4-f869ffca714a" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 23:35:44.046971 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:35:44.046937    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:35:44.220507 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:35:44.220475    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-856f4c4b54-grmbw"
Sep 17 23:35:44.221208 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:35:44.221162    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-856f4c4b54-grmbw" podUID="9787d7a5-4cc3-4c33-a5d4-f869ffca714a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Sep 17 23:35:44.221433 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:35:44.221409    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-856f4c4b54-grmbw" podUID="9787d7a5-4cc3-4c33-a5d4-f869ffca714a" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 23:35:54.221432 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:35:54.221390    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-856f4c4b54-grmbw" podUID="9787d7a5-4cc3-4c33-a5d4-f869ffca714a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Sep 17 23:35:54.221943 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:35:54.221855    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-856f4c4b54-grmbw" podUID="9787d7a5-4cc3-4c33-a5d4-f869ffca714a" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 23:35:55.488992 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:35:55.488957    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:36:04.221818 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:36:04.221761    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-856f4c4b54-grmbw" podUID="9787d7a5-4cc3-4c33-a5d4-f869ffca714a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Sep 17 23:36:04.222448 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:36:04.222200    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-856f4c4b54-grmbw" podUID="9787d7a5-4cc3-4c33-a5d4-f869ffca714a" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 23:36:14.221958 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:36:14.221906    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-856f4c4b54-grmbw" podUID="9787d7a5-4cc3-4c33-a5d4-f869ffca714a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Sep 17 23:36:14.222419 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:36:14.222335    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-856f4c4b54-grmbw" podUID="9787d7a5-4cc3-4c33-a5d4-f869ffca714a" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 23:36:24.221282 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:36:24.221235    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-856f4c4b54-grmbw" podUID="9787d7a5-4cc3-4c33-a5d4-f869ffca714a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Sep 17 23:36:24.221757 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:36:24.221733    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-856f4c4b54-grmbw" podUID="9787d7a5-4cc3-4c33-a5d4-f869ffca714a" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 23:36:34.221545 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:36:34.221502    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-856f4c4b54-grmbw" podUID="9787d7a5-4cc3-4c33-a5d4-f869ffca714a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Sep 17 23:36:34.222056 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:36:34.222034    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-856f4c4b54-grmbw" podUID="9787d7a5-4cc3-4c33-a5d4-f869ffca714a" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 23:36:44.222048 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:36:44.221980    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-856f4c4b54-grmbw" podUID="9787d7a5-4cc3-4c33-a5d4-f869ffca714a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Sep 17 23:36:44.222585 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:36:44.222547    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-856f4c4b54-grmbw" podUID="9787d7a5-4cc3-4c33-a5d4-f869ffca714a" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 23:36:54.221712 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:36:54.221679    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-856f4c4b54-grmbw"
Sep 17 23:36:54.222191 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:36:54.221928    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-856f4c4b54-grmbw"
Sep 17 23:37:07.278846 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:37:07.278762    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_message-dumper-raw-cipn-predictor-9774bf55d-6d2jt_d8f4cba2-2fae-4416-aedf-c6ba28bb5385/kserve-container/0.log"
Sep 17 23:37:07.550404 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:37:07.550323    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-856f4c4b54-grmbw"]
Sep 17 23:37:07.550627 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:37:07.550582    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-856f4c4b54-grmbw" podUID="9787d7a5-4cc3-4c33-a5d4-f869ffca714a" containerName="kserve-container" containerID="cri-o://726e8a6056a2475b892ad9f7c8cbf029801142fab4ec3fd6b8f5ff805c36a828" gracePeriod=30
Sep 17 23:37:07.550679 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:37:07.550617    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-856f4c4b54-grmbw" podUID="9787d7a5-4cc3-4c33-a5d4-f869ffca714a" containerName="agent" containerID="cri-o://4826cefc99c2b7fc404f8543ea7015c7794548e4804618c7272691f410f11851" gracePeriod=30
Sep 17 23:37:07.550679 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:37:07.550635    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-856f4c4b54-grmbw" podUID="9787d7a5-4cc3-4c33-a5d4-f869ffca714a" containerName="kube-rbac-proxy" containerID="cri-o://4a1cfd4ef041455a0258b7bdae9e4fd5a3ff10fdab94d8c166ab62510bd35d73" gracePeriod=30
Sep 17 23:37:07.601486 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:37:07.601455    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:37:07.616806 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:37:07.616776    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-6d2jt"]
Sep 17 23:37:07.617023 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:37:07.616984    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-6d2jt" podUID="d8f4cba2-2fae-4416-aedf-c6ba28bb5385" containerName="kserve-container" containerID="cri-o://c3e7e8ac59770882a54c0bd6d2f3be782629af629ca6d091728d30253bea083f" gracePeriod=30
Sep 17 23:37:07.617170 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:37:07.617050    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-6d2jt" podUID="d8f4cba2-2fae-4416-aedf-c6ba28bb5385" containerName="kube-rbac-proxy" containerID="cri-o://f256bccea2348e4371fff5df8bf6a1ce9cd7c9c840ede619fc04d505028b3569" gracePeriod=30
Sep 17 23:37:07.791474 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:37:07.791450    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:37:07.876273 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:37:07.876248    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-6d2jt"
Sep 17 23:37:07.937549 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:37:07.937514    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-shmkb\" (UniqueName: \"kubernetes.io/projected/d8f4cba2-2fae-4416-aedf-c6ba28bb5385-kube-api-access-shmkb\") pod \"d8f4cba2-2fae-4416-aedf-c6ba28bb5385\" (UID: \"d8f4cba2-2fae-4416-aedf-c6ba28bb5385\") "
Sep 17 23:37:07.937747 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:37:07.937596    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d8f4cba2-2fae-4416-aedf-c6ba28bb5385-proxy-tls\") pod \"d8f4cba2-2fae-4416-aedf-c6ba28bb5385\" (UID: \"d8f4cba2-2fae-4416-aedf-c6ba28bb5385\") "
Sep 17 23:37:07.937747 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:37:07.937617    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d8f4cba2-2fae-4416-aedf-c6ba28bb5385-kube-rbac-proxy-sar-config\") pod \"d8f4cba2-2fae-4416-aedf-c6ba28bb5385\" (UID: \"d8f4cba2-2fae-4416-aedf-c6ba28bb5385\") "
Sep 17 23:37:07.938049 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:37:07.938019    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8f4cba2-2fae-4416-aedf-c6ba28bb5385-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "d8f4cba2-2fae-4416-aedf-c6ba28bb5385" (UID: "d8f4cba2-2fae-4416-aedf-c6ba28bb5385"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:37:07.939853 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:37:07.939833    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8f4cba2-2fae-4416-aedf-c6ba28bb5385-kube-api-access-shmkb" (OuterVolumeSpecName: "kube-api-access-shmkb") pod "d8f4cba2-2fae-4416-aedf-c6ba28bb5385" (UID: "d8f4cba2-2fae-4416-aedf-c6ba28bb5385"). InnerVolumeSpecName "kube-api-access-shmkb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:37:07.939948 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:37:07.939850    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d8f4cba2-2fae-4416-aedf-c6ba28bb5385-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d8f4cba2-2fae-4416-aedf-c6ba28bb5385" (UID: "d8f4cba2-2fae-4416-aedf-c6ba28bb5385"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:37:08.038437 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:37:08.038396    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-shmkb\" (UniqueName: \"kubernetes.io/projected/d8f4cba2-2fae-4416-aedf-c6ba28bb5385-kube-api-access-shmkb\") on node \"ip-10-0-142-4.ec2.internal\" DevicePath \"\""
Sep 17 23:37:08.038437 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:37:08.038433    2584 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d8f4cba2-2fae-4416-aedf-c6ba28bb5385-proxy-tls\") on node \"ip-10-0-142-4.ec2.internal\" DevicePath \"\""
Sep 17 23:37:08.038437 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:37:08.038446    2584 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d8f4cba2-2fae-4416-aedf-c6ba28bb5385-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-4.ec2.internal\" DevicePath \"\""
Sep 17 23:37:08.589395 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:37:08.589364    2584 generic.go:356] "Generic (PLEG): container finished" podID="d8f4cba2-2fae-4416-aedf-c6ba28bb5385" containerID="f256bccea2348e4371fff5df8bf6a1ce9cd7c9c840ede619fc04d505028b3569" exitCode=2
Sep 17 23:37:08.589395 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:37:08.589388    2584 generic.go:356] "Generic (PLEG): container finished" podID="d8f4cba2-2fae-4416-aedf-c6ba28bb5385" containerID="c3e7e8ac59770882a54c0bd6d2f3be782629af629ca6d091728d30253bea083f" exitCode=2
Sep 17 23:37:08.589905 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:37:08.589418    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-6d2jt"
Sep 17 23:37:08.589905 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:37:08.589448    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-6d2jt" event={"ID":"d8f4cba2-2fae-4416-aedf-c6ba28bb5385","Type":"ContainerDied","Data":"f256bccea2348e4371fff5df8bf6a1ce9cd7c9c840ede619fc04d505028b3569"}
Sep 17 23:37:08.589905 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:37:08.589510    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-6d2jt" event={"ID":"d8f4cba2-2fae-4416-aedf-c6ba28bb5385","Type":"ContainerDied","Data":"c3e7e8ac59770882a54c0bd6d2f3be782629af629ca6d091728d30253bea083f"}
Sep 17 23:37:08.589905 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:37:08.589543    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-6d2jt" event={"ID":"d8f4cba2-2fae-4416-aedf-c6ba28bb5385","Type":"ContainerDied","Data":"6d00e0df56e37f70b3f8a7f1e488bcedf5479847615d92723ad2072f05d6952c"}
Sep 17 23:37:08.589905 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:37:08.589555    2584 scope.go:117] "RemoveContainer" containerID="f256bccea2348e4371fff5df8bf6a1ce9cd7c9c840ede619fc04d505028b3569"
Sep 17 23:37:08.592299 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:37:08.592279    2584 generic.go:356] "Generic (PLEG): container finished" podID="9787d7a5-4cc3-4c33-a5d4-f869ffca714a" containerID="4a1cfd4ef041455a0258b7bdae9e4fd5a3ff10fdab94d8c166ab62510bd35d73" exitCode=2
Sep 17 23:37:08.592414 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:37:08.592316    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-856f4c4b54-grmbw" event={"ID":"9787d7a5-4cc3-4c33-a5d4-f869ffca714a","Type":"ContainerDied","Data":"4a1cfd4ef041455a0258b7bdae9e4fd5a3ff10fdab94d8c166ab62510bd35d73"}
Sep 17 23:37:08.599075 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:37:08.599033    2584 scope.go:117] "RemoveContainer" containerID="c3e7e8ac59770882a54c0bd6d2f3be782629af629ca6d091728d30253bea083f"
Sep 17 23:37:08.606974 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:37:08.606957    2584 scope.go:117] "RemoveContainer" containerID="f256bccea2348e4371fff5df8bf6a1ce9cd7c9c840ede619fc04d505028b3569"
Sep 17 23:37:08.607346 ip-10-0-142-4 kubenswrapper[2584]: E0917 23:37:08.607325    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f256bccea2348e4371fff5df8bf6a1ce9cd7c9c840ede619fc04d505028b3569\": container with ID starting with f256bccea2348e4371fff5df8bf6a1ce9cd7c9c840ede619fc04d505028b3569 not found: ID does not exist" containerID="f256bccea2348e4371fff5df8bf6a1ce9cd7c9c840ede619fc04d505028b3569"
Sep 17 23:37:08.607438 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:37:08.607357    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f256bccea2348e4371fff5df8bf6a1ce9cd7c9c840ede619fc04d505028b3569"} err="failed to get container status \"f256bccea2348e4371fff5df8bf6a1ce9cd7c9c840ede619fc04d505028b3569\": rpc error: code = NotFound desc = could not find container \"f256bccea2348e4371fff5df8bf6a1ce9cd7c9c840ede619fc04d505028b3569\": container with ID starting with f256bccea2348e4371fff5df8bf6a1ce9cd7c9c840ede619fc04d505028b3569 not found: ID does not exist"
Sep 17 23:37:08.607438 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:37:08.607376    2584 scope.go:117] "RemoveContainer" containerID="c3e7e8ac59770882a54c0bd6d2f3be782629af629ca6d091728d30253bea083f"
Sep 17 23:37:08.607670 ip-10-0-142-4 kubenswrapper[2584]: E0917 23:37:08.607648    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c3e7e8ac59770882a54c0bd6d2f3be782629af629ca6d091728d30253bea083f\": container with ID starting with c3e7e8ac59770882a54c0bd6d2f3be782629af629ca6d091728d30253bea083f not found: ID does not exist" containerID="c3e7e8ac59770882a54c0bd6d2f3be782629af629ca6d091728d30253bea083f"
Sep 17 23:37:08.607748 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:37:08.607685    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3e7e8ac59770882a54c0bd6d2f3be782629af629ca6d091728d30253bea083f"} err="failed to get container status \"c3e7e8ac59770882a54c0bd6d2f3be782629af629ca6d091728d30253bea083f\": rpc error: code = NotFound desc = could not find container \"c3e7e8ac59770882a54c0bd6d2f3be782629af629ca6d091728d30253bea083f\": container with ID starting with c3e7e8ac59770882a54c0bd6d2f3be782629af629ca6d091728d30253bea083f not found: ID does not exist"
Sep 17 23:37:08.607748 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:37:08.607707    2584 scope.go:117] "RemoveContainer" containerID="f256bccea2348e4371fff5df8bf6a1ce9cd7c9c840ede619fc04d505028b3569"
Sep 17 23:37:08.607941 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:37:08.607922    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f256bccea2348e4371fff5df8bf6a1ce9cd7c9c840ede619fc04d505028b3569"} err="failed to get container status \"f256bccea2348e4371fff5df8bf6a1ce9cd7c9c840ede619fc04d505028b3569\": rpc error: code = NotFound desc = could not find container \"f256bccea2348e4371fff5df8bf6a1ce9cd7c9c840ede619fc04d505028b3569\": container with ID starting with f256bccea2348e4371fff5df8bf6a1ce9cd7c9c840ede619fc04d505028b3569 not found: ID does not exist"
Sep 17 23:37:08.607986 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:37:08.607942    2584 scope.go:117] "RemoveContainer" containerID="c3e7e8ac59770882a54c0bd6d2f3be782629af629ca6d091728d30253bea083f"
Sep 17 23:37:08.608196 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:37:08.608177    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3e7e8ac59770882a54c0bd6d2f3be782629af629ca6d091728d30253bea083f"} err="failed to get container status \"c3e7e8ac59770882a54c0bd6d2f3be782629af629ca6d091728d30253bea083f\": rpc error: code = NotFound desc = could not find container \"c3e7e8ac59770882a54c0bd6d2f3be782629af629ca6d091728d30253bea083f\": container with ID starting with c3e7e8ac59770882a54c0bd6d2f3be782629af629ca6d091728d30253bea083f not found: ID does not exist"
Sep 17 23:37:08.613901 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:37:08.613878    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-6d2jt"]
Sep 17 23:37:08.618083 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:37:08.618060    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-6d2jt"]
Sep 17 23:37:08.765287 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:37:08.765256    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d8f4cba2-2fae-4416-aedf-c6ba28bb5385" path="/var/lib/kubelet/pods/d8f4cba2-2fae-4416-aedf-c6ba28bb5385/volumes"
Sep 17 23:37:08.801122 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:37:08.801084    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-7gr5m/must-gather-f7pnh"]
Sep 17 23:37:08.801443 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:37:08.801426    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d8f4cba2-2fae-4416-aedf-c6ba28bb5385" containerName="kube-rbac-proxy"
Sep 17 23:37:08.801443 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:37:08.801443    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="d8f4cba2-2fae-4416-aedf-c6ba28bb5385" containerName="kube-rbac-proxy"
Sep 17 23:37:08.801590 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:37:08.801455    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d8f4cba2-2fae-4416-aedf-c6ba28bb5385" containerName="kserve-container"
Sep 17 23:37:08.801590 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:37:08.801460    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="d8f4cba2-2fae-4416-aedf-c6ba28bb5385" containerName="kserve-container"
Sep 17 23:37:08.801590 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:37:08.801519    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="d8f4cba2-2fae-4416-aedf-c6ba28bb5385" containerName="kserve-container"
Sep 17 23:37:08.801590 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:37:08.801529    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="d8f4cba2-2fae-4416-aedf-c6ba28bb5385" containerName="kube-rbac-proxy"
Sep 17 23:37:08.805691 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:37:08.805674    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-7gr5m/must-gather-f7pnh"
Sep 17 23:37:08.808447 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:37:08.808418    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-7gr5m\"/\"default-dockercfg-4xcn4\""
Sep 17 23:37:08.808583 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:37:08.808456    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-7gr5m\"/\"kube-root-ca.crt\""
Sep 17 23:37:08.808583 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:37:08.808497    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-7gr5m\"/\"openshift-service-ca.crt\""
Sep 17 23:37:08.821110 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:37:08.821081    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-7gr5m/must-gather-f7pnh"]
Sep 17 23:37:08.829544 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:37:08.829511    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:37:08.844774 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:37:08.844748    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/e1ed49b8-e5f6-4c72-b9d8-fc6c78b4c255-must-gather-output\") pod \"must-gather-f7pnh\" (UID: \"e1ed49b8-e5f6-4c72-b9d8-fc6c78b4c255\") " pod="openshift-must-gather-7gr5m/must-gather-f7pnh"
Sep 17 23:37:08.844884 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:37:08.844814    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-99zqp\" (UniqueName: \"kubernetes.io/projected/e1ed49b8-e5f6-4c72-b9d8-fc6c78b4c255-kube-api-access-99zqp\") pod \"must-gather-f7pnh\" (UID: \"e1ed49b8-e5f6-4c72-b9d8-fc6c78b4c255\") " pod="openshift-must-gather-7gr5m/must-gather-f7pnh"
Sep 17 23:37:08.927694 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:37:08.927650    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:37:08.946130 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:37:08.946099    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/e1ed49b8-e5f6-4c72-b9d8-fc6c78b4c255-must-gather-output\") pod \"must-gather-f7pnh\" (UID: \"e1ed49b8-e5f6-4c72-b9d8-fc6c78b4c255\") " pod="openshift-must-gather-7gr5m/must-gather-f7pnh"
Sep 17 23:37:08.946272 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:37:08.946159    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-99zqp\" (UniqueName: \"kubernetes.io/projected/e1ed49b8-e5f6-4c72-b9d8-fc6c78b4c255-kube-api-access-99zqp\") pod \"must-gather-f7pnh\" (UID: \"e1ed49b8-e5f6-4c72-b9d8-fc6c78b4c255\") " pod="openshift-must-gather-7gr5m/must-gather-f7pnh"
Sep 17 23:37:08.946482 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:37:08.946458    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/e1ed49b8-e5f6-4c72-b9d8-fc6c78b4c255-must-gather-output\") pod \"must-gather-f7pnh\" (UID: \"e1ed49b8-e5f6-4c72-b9d8-fc6c78b4c255\") " pod="openshift-must-gather-7gr5m/must-gather-f7pnh"
Sep 17 23:37:08.956586 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:37:08.956563    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-99zqp\" (UniqueName: \"kubernetes.io/projected/e1ed49b8-e5f6-4c72-b9d8-fc6c78b4c255-kube-api-access-99zqp\") pod \"must-gather-f7pnh\" (UID: \"e1ed49b8-e5f6-4c72-b9d8-fc6c78b4c255\") " pod="openshift-must-gather-7gr5m/must-gather-f7pnh"
Sep 17 23:37:09.124491 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:37:09.124406    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-7gr5m/must-gather-f7pnh"
Sep 17 23:37:09.216609 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:37:09.216575    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-856f4c4b54-grmbw" podUID="9787d7a5-4cc3-4c33-a5d4-f869ffca714a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.32:8643/healthz\": dial tcp 10.132.0.32:8643: connect: connection refused"
Sep 17 23:37:09.399562 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:37:09.399361    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-7gr5m/must-gather-f7pnh"]
Sep 17 23:37:09.402210 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:37:09.402180    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode1ed49b8_e5f6_4c72_b9d8_fc6c78b4c255.slice/crio-a4c6f62531026b30f03700588b8d23513e5c9f518360cd9dd08e7eecdd9e19e0 WatchSource:0}: Error finding container a4c6f62531026b30f03700588b8d23513e5c9f518360cd9dd08e7eecdd9e19e0: Status 404 returned error can't find the container with id a4c6f62531026b30f03700588b8d23513e5c9f518360cd9dd08e7eecdd9e19e0
Sep 17 23:37:09.599309 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:37:09.599274    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-7gr5m/must-gather-f7pnh" event={"ID":"e1ed49b8-e5f6-4c72-b9d8-fc6c78b4c255","Type":"ContainerStarted","Data":"a4c6f62531026b30f03700588b8d23513e5c9f518360cd9dd08e7eecdd9e19e0"}
Sep 17 23:37:13.628986 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:37:13.628896    2584 generic.go:356] "Generic (PLEG): container finished" podID="9787d7a5-4cc3-4c33-a5d4-f869ffca714a" containerID="726e8a6056a2475b892ad9f7c8cbf029801142fab4ec3fd6b8f5ff805c36a828" exitCode=0
Sep 17 23:37:13.628986 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:37:13.628971    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-856f4c4b54-grmbw" event={"ID":"9787d7a5-4cc3-4c33-a5d4-f869ffca714a","Type":"ContainerDied","Data":"726e8a6056a2475b892ad9f7c8cbf029801142fab4ec3fd6b8f5ff805c36a828"}
Sep 17 23:37:14.216563 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:37:14.216517    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-856f4c4b54-grmbw" podUID="9787d7a5-4cc3-4c33-a5d4-f869ffca714a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.32:8643/healthz\": dial tcp 10.132.0.32:8643: connect: connection refused"
Sep 17 23:37:14.222193 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:37:14.222147    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-856f4c4b54-grmbw" podUID="9787d7a5-4cc3-4c33-a5d4-f869ffca714a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Sep 17 23:37:14.222466 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:37:14.222427    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-856f4c4b54-grmbw" podUID="9787d7a5-4cc3-4c33-a5d4-f869ffca714a" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 23:37:16.647102 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:37:16.647063    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-7gr5m/must-gather-f7pnh" event={"ID":"e1ed49b8-e5f6-4c72-b9d8-fc6c78b4c255","Type":"ContainerStarted","Data":"ff32d262e4eae6682763d797c98410c0e7514b2518224adaf1b516d66e865a66"}
Sep 17 23:37:16.647545 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:37:16.647119    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-7gr5m/must-gather-f7pnh" event={"ID":"e1ed49b8-e5f6-4c72-b9d8-fc6c78b4c255","Type":"ContainerStarted","Data":"90f1b302e770872cf3e7e8c34e84965a15325766b7cbf37e5c50a19093bd12fe"}
Sep 17 23:37:16.664234 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:37:16.664165    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-7gr5m/must-gather-f7pnh" podStartSLOduration=2.407256302 podStartE2EDuration="8.664144273s" totalImagesPullingTime="6.256887971s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:37:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:37:16.663878152 +0000 UTC m=+1620.029538062" watchObservedRunningTime="2026-09-17 23:37:16.664144273 +0000 UTC m=+1620.029804191"
Sep 17 23:37:19.216626 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:37:19.216540    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-856f4c4b54-grmbw" podUID="9787d7a5-4cc3-4c33-a5d4-f869ffca714a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.32:8643/healthz\": dial tcp 10.132.0.32:8643: connect: connection refused"
Sep 17 23:37:19.217081 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:37:19.216678    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-856f4c4b54-grmbw"
Sep 17 23:37:24.216717 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:37:24.216670    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-856f4c4b54-grmbw" podUID="9787d7a5-4cc3-4c33-a5d4-f869ffca714a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.32:8643/healthz\": dial tcp 10.132.0.32:8643: connect: connection refused"
Sep 17 23:37:24.221571 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:37:24.221526    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-856f4c4b54-grmbw" podUID="9787d7a5-4cc3-4c33-a5d4-f869ffca714a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Sep 17 23:37:24.221964 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:37:24.221924    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-856f4c4b54-grmbw" podUID="9787d7a5-4cc3-4c33-a5d4-f869ffca714a" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 23:37:24.775865 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:37:24.775835    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-856f4c4b54-grmbw_9787d7a5-4cc3-4c33-a5d4-f869ffca714a/kserve-container/0.log"
Sep 17 23:37:24.802508 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:37:24.802481    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-856f4c4b54-grmbw_9787d7a5-4cc3-4c33-a5d4-f869ffca714a/kube-rbac-proxy/0.log"
Sep 17 23:37:24.817481 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:37:24.817446    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-856f4c4b54-grmbw_9787d7a5-4cc3-4c33-a5d4-f869ffca714a/agent/0.log"
Sep 17 23:37:24.831402 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:37:24.831370    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-856f4c4b54-grmbw_9787d7a5-4cc3-4c33-a5d4-f869ffca714a/storage-initializer/0.log"
Sep 17 23:37:25.690940 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:37:25.690912    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-856f4c4b54-grmbw_9787d7a5-4cc3-4c33-a5d4-f869ffca714a/kserve-container/0.log"
Sep 17 23:37:25.707075 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:37:25.707040    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-856f4c4b54-grmbw_9787d7a5-4cc3-4c33-a5d4-f869ffca714a/kube-rbac-proxy/0.log"
Sep 17 23:37:25.721441 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:37:25.721416    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-856f4c4b54-grmbw_9787d7a5-4cc3-4c33-a5d4-f869ffca714a/agent/0.log"
Sep 17 23:37:25.735108 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:37:25.735081    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-856f4c4b54-grmbw_9787d7a5-4cc3-4c33-a5d4-f869ffca714a/storage-initializer/0.log"
Sep 17 23:37:26.607132 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:37:26.607108    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-856f4c4b54-grmbw_9787d7a5-4cc3-4c33-a5d4-f869ffca714a/kserve-container/0.log"
Sep 17 23:37:26.620374 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:37:26.620353    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-856f4c4b54-grmbw_9787d7a5-4cc3-4c33-a5d4-f869ffca714a/kube-rbac-proxy/0.log"
Sep 17 23:37:26.631840 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:37:26.631818    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-856f4c4b54-grmbw_9787d7a5-4cc3-4c33-a5d4-f869ffca714a/agent/0.log"
Sep 17 23:37:26.644564 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:37:26.644548    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-856f4c4b54-grmbw_9787d7a5-4cc3-4c33-a5d4-f869ffca714a/storage-initializer/0.log"
Sep 17 23:37:27.542098 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:37:27.542069    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-856f4c4b54-grmbw_9787d7a5-4cc3-4c33-a5d4-f869ffca714a/kserve-container/0.log"
Sep 17 23:37:27.573683 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:37:27.573655    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-856f4c4b54-grmbw_9787d7a5-4cc3-4c33-a5d4-f869ffca714a/kube-rbac-proxy/0.log"
Sep 17 23:37:27.602169 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:37:27.602136    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-856f4c4b54-grmbw_9787d7a5-4cc3-4c33-a5d4-f869ffca714a/agent/0.log"
Sep 17 23:37:27.636261 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:37:27.636234    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-856f4c4b54-grmbw_9787d7a5-4cc3-4c33-a5d4-f869ffca714a/storage-initializer/0.log"
Sep 17 23:37:28.635270 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:37:28.635241    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-856f4c4b54-grmbw_9787d7a5-4cc3-4c33-a5d4-f869ffca714a/kserve-container/0.log"
Sep 17 23:37:28.650941 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:37:28.650913    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-856f4c4b54-grmbw_9787d7a5-4cc3-4c33-a5d4-f869ffca714a/kube-rbac-proxy/0.log"
Sep 17 23:37:28.661189 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:37:28.661146    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-856f4c4b54-grmbw_9787d7a5-4cc3-4c33-a5d4-f869ffca714a/agent/0.log"
Sep 17 23:37:28.673542 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:37:28.673517    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-856f4c4b54-grmbw_9787d7a5-4cc3-4c33-a5d4-f869ffca714a/storage-initializer/0.log"
Sep 17 23:37:29.217255 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:37:29.217207    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-856f4c4b54-grmbw" podUID="9787d7a5-4cc3-4c33-a5d4-f869ffca714a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.32:8643/healthz\": dial tcp 10.132.0.32:8643: connect: connection refused"
Sep 17 23:37:29.523582 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:37:29.523548    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-856f4c4b54-grmbw_9787d7a5-4cc3-4c33-a5d4-f869ffca714a/kserve-container/0.log"
Sep 17 23:37:29.538301 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:37:29.538270    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-856f4c4b54-grmbw_9787d7a5-4cc3-4c33-a5d4-f869ffca714a/kube-rbac-proxy/0.log"
Sep 17 23:37:29.549247 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:37:29.549220    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-856f4c4b54-grmbw_9787d7a5-4cc3-4c33-a5d4-f869ffca714a/agent/0.log"
Sep 17 23:37:29.561577 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:37:29.561557    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-856f4c4b54-grmbw_9787d7a5-4cc3-4c33-a5d4-f869ffca714a/storage-initializer/0.log"
Sep 17 23:37:30.381348 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:37:30.381317    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-856f4c4b54-grmbw_9787d7a5-4cc3-4c33-a5d4-f869ffca714a/kserve-container/0.log"
Sep 17 23:37:30.394491 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:37:30.394460    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-856f4c4b54-grmbw_9787d7a5-4cc3-4c33-a5d4-f869ffca714a/kube-rbac-proxy/0.log"
Sep 17 23:37:30.405964 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:37:30.405926    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-856f4c4b54-grmbw_9787d7a5-4cc3-4c33-a5d4-f869ffca714a/agent/0.log"
Sep 17 23:37:30.421242 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:37:30.421215    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-856f4c4b54-grmbw_9787d7a5-4cc3-4c33-a5d4-f869ffca714a/storage-initializer/0.log"
Sep 17 23:37:31.258691 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:37:31.258640    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-856f4c4b54-grmbw_9787d7a5-4cc3-4c33-a5d4-f869ffca714a/kserve-container/0.log"
Sep 17 23:37:31.280291 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:37:31.280264    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-856f4c4b54-grmbw_9787d7a5-4cc3-4c33-a5d4-f869ffca714a/kube-rbac-proxy/0.log"
Sep 17 23:37:31.297665 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:37:31.297640    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-856f4c4b54-grmbw_9787d7a5-4cc3-4c33-a5d4-f869ffca714a/agent/0.log"
Sep 17 23:37:31.317461 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:37:31.317420    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-856f4c4b54-grmbw_9787d7a5-4cc3-4c33-a5d4-f869ffca714a/storage-initializer/0.log"
Sep 17 23:37:32.251259 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:37:32.251232    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-856f4c4b54-grmbw_9787d7a5-4cc3-4c33-a5d4-f869ffca714a/kserve-container/0.log"
Sep 17 23:37:32.273292 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:37:32.273263    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-856f4c4b54-grmbw_9787d7a5-4cc3-4c33-a5d4-f869ffca714a/kube-rbac-proxy/0.log"
Sep 17 23:37:32.295784 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:37:32.295743    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-856f4c4b54-grmbw_9787d7a5-4cc3-4c33-a5d4-f869ffca714a/agent/0.log"
Sep 17 23:37:32.321931 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:37:32.321906    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-856f4c4b54-grmbw_9787d7a5-4cc3-4c33-a5d4-f869ffca714a/storage-initializer/0.log"
Sep 17 23:37:33.238068 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:37:33.238043    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-856f4c4b54-grmbw_9787d7a5-4cc3-4c33-a5d4-f869ffca714a/kserve-container/0.log"
Sep 17 23:37:33.255906 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:37:33.255882    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-856f4c4b54-grmbw_9787d7a5-4cc3-4c33-a5d4-f869ffca714a/kube-rbac-proxy/0.log"
Sep 17 23:37:33.278994 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:37:33.278967    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-856f4c4b54-grmbw_9787d7a5-4cc3-4c33-a5d4-f869ffca714a/agent/0.log"
Sep 17 23:37:33.300047 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:37:33.299994    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-856f4c4b54-grmbw_9787d7a5-4cc3-4c33-a5d4-f869ffca714a/storage-initializer/0.log"
Sep 17 23:37:34.168565 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:37:34.168534    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-856f4c4b54-grmbw_9787d7a5-4cc3-4c33-a5d4-f869ffca714a/kserve-container/0.log"
Sep 17 23:37:34.185190 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:37:34.185148    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-856f4c4b54-grmbw_9787d7a5-4cc3-4c33-a5d4-f869ffca714a/kube-rbac-proxy/0.log"
Sep 17 23:37:34.202983 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:37:34.202954    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-856f4c4b54-grmbw_9787d7a5-4cc3-4c33-a5d4-f869ffca714a/agent/0.log"
Sep 17 23:37:34.216195 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:37:34.216166    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-856f4c4b54-grmbw_9787d7a5-4cc3-4c33-a5d4-f869ffca714a/storage-initializer/0.log"
Sep 17 23:37:34.216379 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:37:34.216356    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-856f4c4b54-grmbw" podUID="9787d7a5-4cc3-4c33-a5d4-f869ffca714a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.32:8643/healthz\": dial tcp 10.132.0.32:8643: connect: connection refused"
Sep 17 23:37:34.221719 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:37:34.221693    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-856f4c4b54-grmbw" podUID="9787d7a5-4cc3-4c33-a5d4-f869ffca714a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Sep 17 23:37:34.221860 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:37:34.221824    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-856f4c4b54-grmbw"
Sep 17 23:37:34.222085 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:37:34.222056    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-856f4c4b54-grmbw" podUID="9787d7a5-4cc3-4c33-a5d4-f869ffca714a" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 23:37:34.222197 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:37:34.222181    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-856f4c4b54-grmbw"
Sep 17 23:37:35.207688 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:37:35.207659    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-856f4c4b54-grmbw_9787d7a5-4cc3-4c33-a5d4-f869ffca714a/kserve-container/0.log"
Sep 17 23:37:35.228706 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:37:35.228679    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-856f4c4b54-grmbw_9787d7a5-4cc3-4c33-a5d4-f869ffca714a/kube-rbac-proxy/0.log"
Sep 17 23:37:35.248996 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:37:35.248961    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-856f4c4b54-grmbw_9787d7a5-4cc3-4c33-a5d4-f869ffca714a/agent/0.log"
Sep 17 23:37:35.263244 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:37:35.263217    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-856f4c4b54-grmbw_9787d7a5-4cc3-4c33-a5d4-f869ffca714a/storage-initializer/0.log"
Sep 17 23:37:36.224548 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:37:36.224521    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-856f4c4b54-grmbw_9787d7a5-4cc3-4c33-a5d4-f869ffca714a/kserve-container/0.log"
Sep 17 23:37:36.244710 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:37:36.244678    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-856f4c4b54-grmbw_9787d7a5-4cc3-4c33-a5d4-f869ffca714a/kube-rbac-proxy/0.log"
Sep 17 23:37:36.261768 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:37:36.261737    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-856f4c4b54-grmbw_9787d7a5-4cc3-4c33-a5d4-f869ffca714a/agent/0.log"
Sep 17 23:37:36.276863 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:37:36.276839    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-856f4c4b54-grmbw_9787d7a5-4cc3-4c33-a5d4-f869ffca714a/storage-initializer/0.log"
Sep 17 23:37:37.391544 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:37:37.391514    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-856f4c4b54-grmbw_9787d7a5-4cc3-4c33-a5d4-f869ffca714a/kserve-container/0.log"
Sep 17 23:37:37.421283 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:37:37.421250    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-856f4c4b54-grmbw_9787d7a5-4cc3-4c33-a5d4-f869ffca714a/kube-rbac-proxy/0.log"
Sep 17 23:37:37.445514 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:37:37.445485    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-856f4c4b54-grmbw_9787d7a5-4cc3-4c33-a5d4-f869ffca714a/agent/0.log"
Sep 17 23:37:37.480159 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:37:37.480130    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-856f4c4b54-grmbw_9787d7a5-4cc3-4c33-a5d4-f869ffca714a/storage-initializer/0.log"
Sep 17 23:37:37.743956 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:37:37.743852    2584 generic.go:356] "Generic (PLEG): container finished" podID="9787d7a5-4cc3-4c33-a5d4-f869ffca714a" containerID="4826cefc99c2b7fc404f8543ea7015c7794548e4804618c7272691f410f11851" exitCode=0
Sep 17 23:37:37.743956 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:37:37.743926    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-856f4c4b54-grmbw" event={"ID":"9787d7a5-4cc3-4c33-a5d4-f869ffca714a","Type":"ContainerDied","Data":"4826cefc99c2b7fc404f8543ea7015c7794548e4804618c7272691f410f11851"}
Sep 17 23:37:38.214351 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:37:38.214315    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-856f4c4b54-grmbw"
Sep 17 23:37:38.315148 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:37:38.315115    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9787d7a5-4cc3-4c33-a5d4-f869ffca714a-proxy-tls\") pod \"9787d7a5-4cc3-4c33-a5d4-f869ffca714a\" (UID: \"9787d7a5-4cc3-4c33-a5d4-f869ffca714a\") "
Sep 17 23:37:38.315345 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:37:38.315176    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9787d7a5-4cc3-4c33-a5d4-f869ffca714a-kube-rbac-proxy-sar-config\") pod \"9787d7a5-4cc3-4c33-a5d4-f869ffca714a\" (UID: \"9787d7a5-4cc3-4c33-a5d4-f869ffca714a\") "
Sep 17 23:37:38.315345 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:37:38.315201    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rztnd\" (UniqueName: \"kubernetes.io/projected/9787d7a5-4cc3-4c33-a5d4-f869ffca714a-kube-api-access-rztnd\") pod \"9787d7a5-4cc3-4c33-a5d4-f869ffca714a\" (UID: \"9787d7a5-4cc3-4c33-a5d4-f869ffca714a\") "
Sep 17 23:37:38.315345 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:37:38.315227    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9787d7a5-4cc3-4c33-a5d4-f869ffca714a-kserve-provision-location\") pod \"9787d7a5-4cc3-4c33-a5d4-f869ffca714a\" (UID: \"9787d7a5-4cc3-4c33-a5d4-f869ffca714a\") "
Sep 17 23:37:38.315345 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:37:38.315279    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9787d7a5-4cc3-4c33-a5d4-f869ffca714a-cabundle-cert\") pod \"9787d7a5-4cc3-4c33-a5d4-f869ffca714a\" (UID: \"9787d7a5-4cc3-4c33-a5d4-f869ffca714a\") "
Sep 17 23:37:38.315566 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:37:38.315533    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9787d7a5-4cc3-4c33-a5d4-f869ffca714a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "9787d7a5-4cc3-4c33-a5d4-f869ffca714a" (UID: "9787d7a5-4cc3-4c33-a5d4-f869ffca714a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:37:38.315634 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:37:38.315558    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9787d7a5-4cc3-4c33-a5d4-f869ffca714a-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "9787d7a5-4cc3-4c33-a5d4-f869ffca714a" (UID: "9787d7a5-4cc3-4c33-a5d4-f869ffca714a"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:37:38.315634 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:37:38.315609    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9787d7a5-4cc3-4c33-a5d4-f869ffca714a-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "9787d7a5-4cc3-4c33-a5d4-f869ffca714a" (UID: "9787d7a5-4cc3-4c33-a5d4-f869ffca714a"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:37:38.317595 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:37:38.317563    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9787d7a5-4cc3-4c33-a5d4-f869ffca714a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "9787d7a5-4cc3-4c33-a5d4-f869ffca714a" (UID: "9787d7a5-4cc3-4c33-a5d4-f869ffca714a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:37:38.317595 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:37:38.317576    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9787d7a5-4cc3-4c33-a5d4-f869ffca714a-kube-api-access-rztnd" (OuterVolumeSpecName: "kube-api-access-rztnd") pod "9787d7a5-4cc3-4c33-a5d4-f869ffca714a" (UID: "9787d7a5-4cc3-4c33-a5d4-f869ffca714a"). InnerVolumeSpecName "kube-api-access-rztnd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:37:38.389946 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:37:38.389913    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-856f4c4b54-grmbw_9787d7a5-4cc3-4c33-a5d4-f869ffca714a/kserve-container/0.log"
Sep 17 23:37:38.406767 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:37:38.406734    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-856f4c4b54-grmbw_9787d7a5-4cc3-4c33-a5d4-f869ffca714a/kube-rbac-proxy/0.log"
Sep 17 23:37:38.415900 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:37:38.415864    2584 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9787d7a5-4cc3-4c33-a5d4-f869ffca714a-proxy-tls\") on node \"ip-10-0-142-4.ec2.internal\" DevicePath \"\""
Sep 17 23:37:38.415900 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:37:38.415896    2584 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9787d7a5-4cc3-4c33-a5d4-f869ffca714a-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-4.ec2.internal\" DevicePath \"\""
Sep 17 23:37:38.416126 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:37:38.415912    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rztnd\" (UniqueName: \"kubernetes.io/projected/9787d7a5-4cc3-4c33-a5d4-f869ffca714a-kube-api-access-rztnd\") on node \"ip-10-0-142-4.ec2.internal\" DevicePath \"\""
Sep 17 23:37:38.416126 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:37:38.415921    2584 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9787d7a5-4cc3-4c33-a5d4-f869ffca714a-kserve-provision-location\") on node \"ip-10-0-142-4.ec2.internal\" DevicePath \"\""
Sep 17 23:37:38.416126 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:37:38.415931    2584 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9787d7a5-4cc3-4c33-a5d4-f869ffca714a-cabundle-cert\") on node \"ip-10-0-142-4.ec2.internal\" DevicePath \"\""
Sep 17 23:37:38.425511 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:37:38.425466    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-856f4c4b54-grmbw_9787d7a5-4cc3-4c33-a5d4-f869ffca714a/agent/0.log"
Sep 17 23:37:38.443349 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:37:38.443321    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-856f4c4b54-grmbw_9787d7a5-4cc3-4c33-a5d4-f869ffca714a/storage-initializer/0.log"
Sep 17 23:37:38.750879 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:37:38.750844    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-856f4c4b54-grmbw" event={"ID":"9787d7a5-4cc3-4c33-a5d4-f869ffca714a","Type":"ContainerDied","Data":"8b3aeaa88f7f683ebd68f47a26f82e8f7f255b410493478472e120931dd972f8"}
Sep 17 23:37:38.751076 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:37:38.750904    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-856f4c4b54-grmbw"
Sep 17 23:37:38.751076 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:37:38.750939    2584 scope.go:117] "RemoveContainer" containerID="4826cefc99c2b7fc404f8543ea7015c7794548e4804618c7272691f410f11851"
Sep 17 23:37:38.764848 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:37:38.764827    2584 scope.go:117] "RemoveContainer" containerID="4a1cfd4ef041455a0258b7bdae9e4fd5a3ff10fdab94d8c166ab62510bd35d73"
Sep 17 23:37:38.773054 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:37:38.773035    2584 scope.go:117] "RemoveContainer" containerID="726e8a6056a2475b892ad9f7c8cbf029801142fab4ec3fd6b8f5ff805c36a828"
Sep 17 23:37:38.783598 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:37:38.783563    2584 scope.go:117] "RemoveContainer" containerID="f49f84bbce952dd3b6fe41ee0d45677ede2b44afb75dfa4ef707afa4546be944"
Sep 17 23:37:38.785746 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:37:38.785682    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-856f4c4b54-grmbw"]
Sep 17 23:37:38.787144 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:37:38.787118    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-856f4c4b54-grmbw"]
Sep 17 23:37:39.758310 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:37:39.758276    2584 generic.go:356] "Generic (PLEG): container finished" podID="e1ed49b8-e5f6-4c72-b9d8-fc6c78b4c255" containerID="90f1b302e770872cf3e7e8c34e84965a15325766b7cbf37e5c50a19093bd12fe" exitCode=0
Sep 17 23:37:39.758808 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:37:39.758346    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-7gr5m/must-gather-f7pnh" event={"ID":"e1ed49b8-e5f6-4c72-b9d8-fc6c78b4c255","Type":"ContainerDied","Data":"90f1b302e770872cf3e7e8c34e84965a15325766b7cbf37e5c50a19093bd12fe"}
Sep 17 23:37:39.758808 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:37:39.758682    2584 scope.go:117] "RemoveContainer" containerID="90f1b302e770872cf3e7e8c34e84965a15325766b7cbf37e5c50a19093bd12fe"
Sep 17 23:37:40.205175 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:37:40.205083    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-7gr5m_must-gather-f7pnh_e1ed49b8-e5f6-4c72-b9d8-fc6c78b4c255/gather/0.log"
Sep 17 23:37:40.677601 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:37:40.677570    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:37:40.762907 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:37:40.762874    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9787d7a5-4cc3-4c33-a5d4-f869ffca714a" path="/var/lib/kubelet/pods/9787d7a5-4cc3-4c33-a5d4-f869ffca714a/volumes"
Sep 17 23:37:40.859397 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:37:40.859367    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-wk5sv/must-gather-4k7vp"]
Sep 17 23:37:40.859673 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:37:40.859662    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9787d7a5-4cc3-4c33-a5d4-f869ffca714a" containerName="kserve-container"
Sep 17 23:37:40.859717 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:37:40.859675    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="9787d7a5-4cc3-4c33-a5d4-f869ffca714a" containerName="kserve-container"
Sep 17 23:37:40.859717 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:37:40.859707    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9787d7a5-4cc3-4c33-a5d4-f869ffca714a" containerName="agent"
Sep 17 23:37:40.859717 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:37:40.859715    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="9787d7a5-4cc3-4c33-a5d4-f869ffca714a" containerName="agent"
Sep 17 23:37:40.859820 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:37:40.859728    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9787d7a5-4cc3-4c33-a5d4-f869ffca714a" containerName="storage-initializer"
Sep 17 23:37:40.859820 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:37:40.859735    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="9787d7a5-4cc3-4c33-a5d4-f869ffca714a" containerName="storage-initializer"
Sep 17 23:37:40.859820 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:37:40.859741    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9787d7a5-4cc3-4c33-a5d4-f869ffca714a" containerName="kube-rbac-proxy"
Sep 17 23:37:40.859820 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:37:40.859746    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="9787d7a5-4cc3-4c33-a5d4-f869ffca714a" containerName="kube-rbac-proxy"
Sep 17 23:37:40.859820 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:37:40.859793    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="9787d7a5-4cc3-4c33-a5d4-f869ffca714a" containerName="agent"
Sep 17 23:37:40.859820 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:37:40.859802    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="9787d7a5-4cc3-4c33-a5d4-f869ffca714a" containerName="kserve-container"
Sep 17 23:37:40.859820 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:37:40.859809    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="9787d7a5-4cc3-4c33-a5d4-f869ffca714a" containerName="kube-rbac-proxy"
Sep 17 23:37:40.863832 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:37:40.863808    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-wk5sv/must-gather-4k7vp"
Sep 17 23:37:40.866928 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:37:40.866902    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-wk5sv\"/\"kube-root-ca.crt\""
Sep 17 23:37:40.867076 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:37:40.866931    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-wk5sv\"/\"openshift-service-ca.crt\""
Sep 17 23:37:40.867076 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:37:40.866902    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-wk5sv\"/\"default-dockercfg-g7v4k\""
Sep 17 23:37:40.874267 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:37:40.874238    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-wk5sv/must-gather-4k7vp"]
Sep 17 23:37:40.879055 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:37:40.879028    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:37:40.938995 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:37:40.938901    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/5c4cb6fe-5186-4373-9b61-988862329689-must-gather-output\") pod \"must-gather-4k7vp\" (UID: \"5c4cb6fe-5186-4373-9b61-988862329689\") " pod="openshift-must-gather-wk5sv/must-gather-4k7vp"
Sep 17 23:37:40.938995 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:37:40.938963    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wfxzm\" (UniqueName: \"kubernetes.io/projected/5c4cb6fe-5186-4373-9b61-988862329689-kube-api-access-wfxzm\") pod \"must-gather-4k7vp\" (UID: \"5c4cb6fe-5186-4373-9b61-988862329689\") " pod="openshift-must-gather-wk5sv/must-gather-4k7vp"
Sep 17 23:37:40.980483 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:37:40.980452    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:37:41.039760 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:37:41.039714    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wfxzm\" (UniqueName: \"kubernetes.io/projected/5c4cb6fe-5186-4373-9b61-988862329689-kube-api-access-wfxzm\") pod \"must-gather-4k7vp\" (UID: \"5c4cb6fe-5186-4373-9b61-988862329689\") " pod="openshift-must-gather-wk5sv/must-gather-4k7vp"
Sep 17 23:37:41.039945 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:37:41.039815    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/5c4cb6fe-5186-4373-9b61-988862329689-must-gather-output\") pod \"must-gather-4k7vp\" (UID: \"5c4cb6fe-5186-4373-9b61-988862329689\") " pod="openshift-must-gather-wk5sv/must-gather-4k7vp"
Sep 17 23:37:41.040220 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:37:41.040201    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/5c4cb6fe-5186-4373-9b61-988862329689-must-gather-output\") pod \"must-gather-4k7vp\" (UID: \"5c4cb6fe-5186-4373-9b61-988862329689\") " pod="openshift-must-gather-wk5sv/must-gather-4k7vp"
Sep 17 23:37:41.051577 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:37:41.051547    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wfxzm\" (UniqueName: \"kubernetes.io/projected/5c4cb6fe-5186-4373-9b61-988862329689-kube-api-access-wfxzm\") pod \"must-gather-4k7vp\" (UID: \"5c4cb6fe-5186-4373-9b61-988862329689\") " pod="openshift-must-gather-wk5sv/must-gather-4k7vp"
Sep 17 23:37:41.173852 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:37:41.173815    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-wk5sv/must-gather-4k7vp"
Sep 17 23:37:41.447507 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:37:41.447274    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-wk5sv/must-gather-4k7vp"]
Sep 17 23:37:41.450074 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:37:41.450039    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5c4cb6fe_5186_4373_9b61_988862329689.slice/crio-70c06fe21921deb5a349697466bdb0a5c1655c2a6e6b89fd2ba893066d5b65bf WatchSource:0}: Error finding container 70c06fe21921deb5a349697466bdb0a5c1655c2a6e6b89fd2ba893066d5b65bf: Status 404 returned error can't find the container with id 70c06fe21921deb5a349697466bdb0a5c1655c2a6e6b89fd2ba893066d5b65bf
Sep 17 23:37:41.768911 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:37:41.768875    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-wk5sv/must-gather-4k7vp" event={"ID":"5c4cb6fe-5186-4373-9b61-988862329689","Type":"ContainerStarted","Data":"70c06fe21921deb5a349697466bdb0a5c1655c2a6e6b89fd2ba893066d5b65bf"}
Sep 17 23:37:42.776074 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:37:42.776034    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-wk5sv/must-gather-4k7vp" event={"ID":"5c4cb6fe-5186-4373-9b61-988862329689","Type":"ContainerStarted","Data":"deb47a95398e833eda9ce9f3ff269ff2824ada44b93e7c5b3b62f02e3df3c414"}
Sep 17 23:37:42.776074 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:37:42.776075    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-wk5sv/must-gather-4k7vp" event={"ID":"5c4cb6fe-5186-4373-9b61-988862329689","Type":"ContainerStarted","Data":"7268c851cbf4118593b103a6be530db2ae5ef19b3f27881f1c2cb75acc11db81"}
Sep 17 23:37:42.795759 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:37:42.795702    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-wk5sv/must-gather-4k7vp" podStartSLOduration=1.956449793 podStartE2EDuration="2.795685094s" totalImagesPullingTime="839.235301ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:37:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:37:42.79341424 +0000 UTC m=+1646.159074154" watchObservedRunningTime="2026-09-17 23:37:42.795685094 +0000 UTC m=+1646.161345011"
Sep 17 23:37:43.845282 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:37:43.845254    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-24tsm_fa08be4f-e950-4dee-a227-feb39c2ba7e0/global-pull-secret-syncer/0.log"
Sep 17 23:37:44.056735 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:37:44.056700    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log"
Sep 17 23:37:44.222999 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:37:44.222884    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-142-4.ec2.internal_3771e4c220b5a223d0f484f25cdc29e8/haproxy/0.log"
Sep 17 23:37:45.661773 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:37:45.661733    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-must-gather-7gr5m/must-gather-f7pnh"]
Sep 17 23:37:45.662316 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:37:45.662232    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-must-gather-7gr5m/must-gather-f7pnh" podUID="e1ed49b8-e5f6-4c72-b9d8-fc6c78b4c255" containerName="copy" containerID="cri-o://ff32d262e4eae6682763d797c98410c0e7514b2518224adaf1b516d66e865a66" gracePeriod=2
Sep 17 23:37:45.664209 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:37:45.664143    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-must-gather-7gr5m/must-gather-f7pnh"]
Sep 17 23:37:45.664985 ip-10-0-142-4 kubenswrapper[2584]: E0917 23:37:45.664950    2584 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-f7pnh\" is forbidden: User \"system:node:ip-10-0-142-4.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-7gr5m\": no relationship found between node 'ip-10-0-142-4.ec2.internal' and this object" podUID="e1ed49b8-e5f6-4c72-b9d8-fc6c78b4c255" pod="openshift-must-gather-7gr5m/must-gather-f7pnh"
Sep 17 23:37:45.796403 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:37:45.796373    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-7gr5m_must-gather-f7pnh_e1ed49b8-e5f6-4c72-b9d8-fc6c78b4c255/copy/0.log"
Sep 17 23:37:45.797177 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:37:45.797150    2584 generic.go:356] "Generic (PLEG): container finished" podID="e1ed49b8-e5f6-4c72-b9d8-fc6c78b4c255" containerID="ff32d262e4eae6682763d797c98410c0e7514b2518224adaf1b516d66e865a66" exitCode=143
Sep 17 23:37:46.051112 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:37:46.051023    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-7gr5m_must-gather-f7pnh_e1ed49b8-e5f6-4c72-b9d8-fc6c78b4c255/copy/0.log"
Sep 17 23:37:46.054270 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:37:46.054242    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-7gr5m/must-gather-f7pnh"
Sep 17 23:37:46.056614 ip-10-0-142-4 kubenswrapper[2584]: E0917 23:37:46.056573    2584 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-f7pnh\" is forbidden: User \"system:node:ip-10-0-142-4.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-7gr5m\": no relationship found between node 'ip-10-0-142-4.ec2.internal' and this object" podUID="e1ed49b8-e5f6-4c72-b9d8-fc6c78b4c255" pod="openshift-must-gather-7gr5m/must-gather-f7pnh"
Sep 17 23:37:46.088181 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:37:46.083830    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/e1ed49b8-e5f6-4c72-b9d8-fc6c78b4c255-must-gather-output\") pod \"e1ed49b8-e5f6-4c72-b9d8-fc6c78b4c255\" (UID: \"e1ed49b8-e5f6-4c72-b9d8-fc6c78b4c255\") "
Sep 17 23:37:46.088181 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:37:46.083901    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-99zqp\" (UniqueName: \"kubernetes.io/projected/e1ed49b8-e5f6-4c72-b9d8-fc6c78b4c255-kube-api-access-99zqp\") pod \"e1ed49b8-e5f6-4c72-b9d8-fc6c78b4c255\" (UID: \"e1ed49b8-e5f6-4c72-b9d8-fc6c78b4c255\") "
Sep 17 23:37:46.088181 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:37:46.086624    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1ed49b8-e5f6-4c72-b9d8-fc6c78b4c255-must-gather-output" (OuterVolumeSpecName: "must-gather-output") pod "e1ed49b8-e5f6-4c72-b9d8-fc6c78b4c255" (UID: "e1ed49b8-e5f6-4c72-b9d8-fc6c78b4c255"). InnerVolumeSpecName "must-gather-output". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:37:46.089252 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:37:46.089203    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1ed49b8-e5f6-4c72-b9d8-fc6c78b4c255-kube-api-access-99zqp" (OuterVolumeSpecName: "kube-api-access-99zqp") pod "e1ed49b8-e5f6-4c72-b9d8-fc6c78b4c255" (UID: "e1ed49b8-e5f6-4c72-b9d8-fc6c78b4c255"). InnerVolumeSpecName "kube-api-access-99zqp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:37:46.185584 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:37:46.185516    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-99zqp\" (UniqueName: \"kubernetes.io/projected/e1ed49b8-e5f6-4c72-b9d8-fc6c78b4c255-kube-api-access-99zqp\") on node \"ip-10-0-142-4.ec2.internal\" DevicePath \"\""
Sep 17 23:37:46.185584 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:37:46.185556    2584 reconciler_common.go:299] "Volume detached for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/e1ed49b8-e5f6-4c72-b9d8-fc6c78b4c255-must-gather-output\") on node \"ip-10-0-142-4.ec2.internal\" DevicePath \"\""
Sep 17 23:37:46.766662 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:37:46.766617    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e1ed49b8-e5f6-4c72-b9d8-fc6c78b4c255" path="/var/lib/kubelet/pods/e1ed49b8-e5f6-4c72-b9d8-fc6c78b4c255/volumes"
Sep 17 23:37:46.823030 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:37:46.822842    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-7gr5m_must-gather-f7pnh_e1ed49b8-e5f6-4c72-b9d8-fc6c78b4c255/copy/0.log"
Sep 17 23:37:46.826158 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:37:46.823474    2584 scope.go:117] "RemoveContainer" containerID="ff32d262e4eae6682763d797c98410c0e7514b2518224adaf1b516d66e865a66"
Sep 17 23:37:46.826158 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:37:46.823690    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-7gr5m/must-gather-f7pnh"
Sep 17 23:37:46.844905 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:37:46.844880    2584 scope.go:117] "RemoveContainer" containerID="90f1b302e770872cf3e7e8c34e84965a15325766b7cbf37e5c50a19093bd12fe"
Sep 17 23:37:46.970993 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:37:46.970946    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:37:47.098262 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:37:47.098069    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2vzq_05765151-c799-4fe2-9f2e-cf4181e84ffa/konnectivity-agent/0.log" limit=1024
Sep 17 23:37:47.960576 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:37:47.960416    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_9d77d57a-58f9-4d62-baef-c4cb8b62816f/alertmanager/0.log"
Sep 17 23:37:48.016841 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:37:48.016815    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_9d77d57a-58f9-4d62-baef-c4cb8b62816f/config-reloader/0.log"
Sep 17 23:37:48.094975 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:37:48.094943    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_9d77d57a-58f9-4d62-baef-c4cb8b62816f/kube-rbac-proxy-web/0.log"
Sep 17 23:37:48.170152 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:37:48.170079    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_9d77d57a-58f9-4d62-baef-c4cb8b62816f/kube-rbac-proxy/0.log"
Sep 17 23:37:48.288585 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:37:48.288548    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_9d77d57a-58f9-4d62-baef-c4cb8b62816f/kube-rbac-proxy-metric/0.log"
Sep 17 23:37:48.374785 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:37:48.374754    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_9d77d57a-58f9-4d62-baef-c4cb8b62816f/prom-label-proxy/0.log"
Sep 17 23:37:48.462486 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:37:48.462433    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_9d77d57a-58f9-4d62-baef-c4cb8b62816f/init-config-reloader/0.log"
Sep 17 23:37:48.626929 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:37:48.626840    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-84c885658c-xlwlk_ee9d22ab-a36f-4603-bf09-81dd9d2663f0/kube-state-metrics/0.log"
Sep 17 23:37:48.670775 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:37:48.670671    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-84c885658c-xlwlk_ee9d22ab-a36f-4603-bf09-81dd9d2663f0/kube-rbac-proxy-main/0.log"
Sep 17 23:37:48.724359 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:37:48.724328    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-84c885658c-xlwlk_ee9d22ab-a36f-4603-bf09-81dd9d2663f0/kube-rbac-proxy-self/0.log"
Sep 17 23:37:48.860244 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:37:48.860154    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-jfd6q_7b67ca5d-0b03-429d-8145-6d25e2068a89/node-exporter/0.log"
Sep 17 23:37:48.893102 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:37:48.893071    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-jfd6q_7b67ca5d-0b03-429d-8145-6d25e2068a89/kube-rbac-proxy/0.log"
Sep 17 23:37:48.927438 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:37:48.927402    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-jfd6q_7b67ca5d-0b03-429d-8145-6d25e2068a89/init-textfile/0.log"
Sep 17 23:37:49.287168 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:37:49.287140    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_2885d33f-5e94-41ee-aff3-fb77ebcce714/prometheus/0.log"
Sep 17 23:37:49.331053 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:37:49.331019    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_2885d33f-5e94-41ee-aff3-fb77ebcce714/config-reloader/0.log"
Sep 17 23:37:49.370336 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:37:49.370307    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_2885d33f-5e94-41ee-aff3-fb77ebcce714/thanos-sidecar/0.log"
Sep 17 23:37:49.400155 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:37:49.400121    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_2885d33f-5e94-41ee-aff3-fb77ebcce714/kube-rbac-proxy-web/0.log"
Sep 17 23:37:49.428241 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:37:49.428217    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_2885d33f-5e94-41ee-aff3-fb77ebcce714/kube-rbac-proxy/0.log"
Sep 17 23:37:49.456124 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:37:49.456100    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_2885d33f-5e94-41ee-aff3-fb77ebcce714/kube-rbac-proxy-thanos/0.log"
Sep 17 23:37:49.485049 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:37:49.484987    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_2885d33f-5e94-41ee-aff3-fb77ebcce714/init-config-reloader/0.log"
Sep 17 23:37:49.535868 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:37:49.535840    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-7bd5d56d6-r6x7z_6885dc6c-af5e-49f8-8052-688cab953847/prometheus-operator/0.log"
Sep 17 23:37:49.566494 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:37:49.566456    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-7bd5d56d6-r6x7z_6885dc6c-af5e-49f8-8052-688cab953847/kube-rbac-proxy/0.log"
Sep 17 23:37:49.972499 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:37:49.972453    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6c99487fbd-897xp_e2b185f4-e371-4556-a3ae-3eda3d226876/thanos-query/0.log"
Sep 17 23:37:50.010964 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:37:50.010934    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6c99487fbd-897xp_e2b185f4-e371-4556-a3ae-3eda3d226876/kube-rbac-proxy-web/0.log"
Sep 17 23:37:50.042149 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:37:50.042123    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6c99487fbd-897xp_e2b185f4-e371-4556-a3ae-3eda3d226876/kube-rbac-proxy/0.log"
Sep 17 23:37:50.066593 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:37:50.066570    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6c99487fbd-897xp_e2b185f4-e371-4556-a3ae-3eda3d226876/prom-label-proxy/0.log"
Sep 17 23:37:50.093984 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:37:50.093956    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6c99487fbd-897xp_e2b185f4-e371-4556-a3ae-3eda3d226876/kube-rbac-proxy-rules/0.log"
Sep 17 23:37:50.121213 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:37:50.121109    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6c99487fbd-897xp_e2b185f4-e371-4556-a3ae-3eda3d226876/kube-rbac-proxy-metrics/0.log"
Sep 17 23:37:51.982239 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:37:51.982197    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-wk5sv/perf-node-gather-daemonset-7czn5"]
Sep 17 23:37:51.982695 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:37:51.982577    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e1ed49b8-e5f6-4c72-b9d8-fc6c78b4c255" containerName="copy"
Sep 17 23:37:51.982695 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:37:51.982592    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="e1ed49b8-e5f6-4c72-b9d8-fc6c78b4c255" containerName="copy"
Sep 17 23:37:51.982695 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:37:51.982616    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e1ed49b8-e5f6-4c72-b9d8-fc6c78b4c255" containerName="gather"
Sep 17 23:37:51.982695 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:37:51.982621    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="e1ed49b8-e5f6-4c72-b9d8-fc6c78b4c255" containerName="gather"
Sep 17 23:37:51.982695 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:37:51.982678    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="e1ed49b8-e5f6-4c72-b9d8-fc6c78b4c255" containerName="gather"
Sep 17 23:37:51.982695 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:37:51.982691    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="e1ed49b8-e5f6-4c72-b9d8-fc6c78b4c255" containerName="copy"
Sep 17 23:37:51.987448 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:37:51.987415    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-wk5sv/perf-node-gather-daemonset-7czn5"
Sep 17 23:37:51.998128 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:37:51.998101    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-wk5sv/perf-node-gather-daemonset-7czn5"]
Sep 17 23:37:52.044317 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:37:52.044275    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0b53df7e-8b3a-4402-8d88-9684020aacb3-sys\") pod \"perf-node-gather-daemonset-7czn5\" (UID: \"0b53df7e-8b3a-4402-8d88-9684020aacb3\") " pod="openshift-must-gather-wk5sv/perf-node-gather-daemonset-7czn5"
Sep 17 23:37:52.044592 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:37:52.044567    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pfthr\" (UniqueName: \"kubernetes.io/projected/0b53df7e-8b3a-4402-8d88-9684020aacb3-kube-api-access-pfthr\") pod \"perf-node-gather-daemonset-7czn5\" (UID: \"0b53df7e-8b3a-4402-8d88-9684020aacb3\") " pod="openshift-must-gather-wk5sv/perf-node-gather-daemonset-7czn5"
Sep 17 23:37:52.044766 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:37:52.044736    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/0b53df7e-8b3a-4402-8d88-9684020aacb3-proc\") pod \"perf-node-gather-daemonset-7czn5\" (UID: \"0b53df7e-8b3a-4402-8d88-9684020aacb3\") " pod="openshift-must-gather-wk5sv/perf-node-gather-daemonset-7czn5"
Sep 17 23:37:52.044924 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:37:52.044907    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0b53df7e-8b3a-4402-8d88-9684020aacb3-lib-modules\") pod \"perf-node-gather-daemonset-7czn5\" (UID: \"0b53df7e-8b3a-4402-8d88-9684020aacb3\") " pod="openshift-must-gather-wk5sv/perf-node-gather-daemonset-7czn5"
Sep 17 23:37:52.045165 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:37:52.045147    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/0b53df7e-8b3a-4402-8d88-9684020aacb3-podres\") pod \"perf-node-gather-daemonset-7czn5\" (UID: \"0b53df7e-8b3a-4402-8d88-9684020aacb3\") " pod="openshift-must-gather-wk5sv/perf-node-gather-daemonset-7czn5"
Sep 17 23:37:52.146107 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:37:52.146061    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/0b53df7e-8b3a-4402-8d88-9684020aacb3-podres\") pod \"perf-node-gather-daemonset-7czn5\" (UID: \"0b53df7e-8b3a-4402-8d88-9684020aacb3\") " pod="openshift-must-gather-wk5sv/perf-node-gather-daemonset-7czn5"
Sep 17 23:37:52.146297 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:37:52.146153    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0b53df7e-8b3a-4402-8d88-9684020aacb3-sys\") pod \"perf-node-gather-daemonset-7czn5\" (UID: \"0b53df7e-8b3a-4402-8d88-9684020aacb3\") " pod="openshift-must-gather-wk5sv/perf-node-gather-daemonset-7czn5"
Sep 17 23:37:52.146297 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:37:52.146183    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pfthr\" (UniqueName: \"kubernetes.io/projected/0b53df7e-8b3a-4402-8d88-9684020aacb3-kube-api-access-pfthr\") pod \"perf-node-gather-daemonset-7czn5\" (UID: \"0b53df7e-8b3a-4402-8d88-9684020aacb3\") " pod="openshift-must-gather-wk5sv/perf-node-gather-daemonset-7czn5"
Sep 17 23:37:52.146297 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:37:52.146212    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/0b53df7e-8b3a-4402-8d88-9684020aacb3-proc\") pod \"perf-node-gather-daemonset-7czn5\" (UID: \"0b53df7e-8b3a-4402-8d88-9684020aacb3\") " pod="openshift-must-gather-wk5sv/perf-node-gather-daemonset-7czn5"
Sep 17 23:37:52.146297 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:37:52.146264    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/0b53df7e-8b3a-4402-8d88-9684020aacb3-podres\") pod \"perf-node-gather-daemonset-7czn5\" (UID: \"0b53df7e-8b3a-4402-8d88-9684020aacb3\") " pod="openshift-must-gather-wk5sv/perf-node-gather-daemonset-7czn5"
Sep 17 23:37:52.146510 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:37:52.146275    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/0b53df7e-8b3a-4402-8d88-9684020aacb3-proc\") pod \"perf-node-gather-daemonset-7czn5\" (UID: \"0b53df7e-8b3a-4402-8d88-9684020aacb3\") " pod="openshift-must-gather-wk5sv/perf-node-gather-daemonset-7czn5"
Sep 17 23:37:52.146510 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:37:52.146276    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0b53df7e-8b3a-4402-8d88-9684020aacb3-sys\") pod \"perf-node-gather-daemonset-7czn5\" (UID: \"0b53df7e-8b3a-4402-8d88-9684020aacb3\") " pod="openshift-must-gather-wk5sv/perf-node-gather-daemonset-7czn5"
Sep 17 23:37:52.146510 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:37:52.146348    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0b53df7e-8b3a-4402-8d88-9684020aacb3-lib-modules\") pod \"perf-node-gather-daemonset-7czn5\" (UID: \"0b53df7e-8b3a-4402-8d88-9684020aacb3\") " pod="openshift-must-gather-wk5sv/perf-node-gather-daemonset-7czn5"
Sep 17 23:37:52.146510 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:37:52.146492    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0b53df7e-8b3a-4402-8d88-9684020aacb3-lib-modules\") pod \"perf-node-gather-daemonset-7czn5\" (UID: \"0b53df7e-8b3a-4402-8d88-9684020aacb3\") " pod="openshift-must-gather-wk5sv/perf-node-gather-daemonset-7czn5"
Sep 17 23:37:52.156920 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:37:52.155380    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pfthr\" (UniqueName: \"kubernetes.io/projected/0b53df7e-8b3a-4402-8d88-9684020aacb3-kube-api-access-pfthr\") pod \"perf-node-gather-daemonset-7czn5\" (UID: \"0b53df7e-8b3a-4402-8d88-9684020aacb3\") " pod="openshift-must-gather-wk5sv/perf-node-gather-daemonset-7czn5"
Sep 17 23:37:52.274311 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:37:52.274270    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-6594f4c75-c9vsj_72436ae5-af59-40fa-8525-c27cf3209e34/download-server/0.log"
Sep 17 23:37:52.302447 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:37:52.302408    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-wk5sv/perf-node-gather-daemonset-7czn5"
Sep 17 23:37:52.593607 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:37:52.593512    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-wk5sv/perf-node-gather-daemonset-7czn5"]
Sep 17 23:37:52.597202 ip-10-0-142-4 kubenswrapper[2584]: W0917 23:37:52.597176    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod0b53df7e_8b3a_4402_8d88_9684020aacb3.slice/crio-8309c01991eb64a3d21a260024a325ca1c79406f7a0493bfa1adb7b64e3d90a3 WatchSource:0}: Error finding container 8309c01991eb64a3d21a260024a325ca1c79406f7a0493bfa1adb7b64e3d90a3: Status 404 returned error can't find the container with id 8309c01991eb64a3d21a260024a325ca1c79406f7a0493bfa1adb7b64e3d90a3
Sep 17 23:37:52.862948 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:37:52.862828    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-wk5sv/perf-node-gather-daemonset-7czn5" event={"ID":"0b53df7e-8b3a-4402-8d88-9684020aacb3","Type":"ContainerStarted","Data":"cbd4534422a8385ce2ef1e733d954f0d66bfdd7417aa5bce011ac4609268518d"}
Sep 17 23:37:52.862948 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:37:52.862917    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-wk5sv/perf-node-gather-daemonset-7czn5" event={"ID":"0b53df7e-8b3a-4402-8d88-9684020aacb3","Type":"ContainerStarted","Data":"8309c01991eb64a3d21a260024a325ca1c79406f7a0493bfa1adb7b64e3d90a3"}
Sep 17 23:37:52.886457 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:37:52.886386    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-wk5sv/perf-node-gather-daemonset-7czn5" podStartSLOduration=1.886363068 podStartE2EDuration="1.886363068s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 23:37:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:37:52.884042272 +0000 UTC m=+1656.249702187" watchObservedRunningTime="2026-09-17 23:37:52.886363068 +0000 UTC m=+1656.252022986"
Sep 17 23:37:53.868088 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:37:53.868048    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-wk5sv/perf-node-gather-daemonset-7czn5"
Sep 17 23:37:53.947760 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:37:53.947730    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-qtz2c_9572448f-49db-4e3b-a174-f994660e7ef1/dns/0.log"
Sep 17 23:37:53.974702 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:37:53.974678    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-qtz2c_9572448f-49db-4e3b-a174-f994660e7ef1/kube-rbac-proxy/0.log"
Sep 17 23:37:54.001702 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:37:54.001673    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-895t6_fcc7ae4e-ace4-490f-a4e0-1c7c0ea22f2d/dns-node-resolver/0.log"
Sep 17 23:37:54.512908 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:37:54.512882    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-5spz7_04f96656-0988-4bea-a70f-79d1b8445201/node-ca/0.log"
Sep 17 23:37:55.805968 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:37:55.805926    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-bkxmq_35051393-90ef-4a48-b95c-a2cc221b538f/serve-healthcheck-canary/0.log"
Sep 17 23:37:56.521388 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:37:56.521355    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-rxcfk_42b0c2c5-0b9f-4b0c-a782-6df4664999f9/kube-rbac-proxy/0.log"
Sep 17 23:37:56.547226 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:37:56.547194    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-rxcfk_42b0c2c5-0b9f-4b0c-a782-6df4664999f9/exporter/0.log"
Sep 17 23:37:56.582241 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:37:56.582209    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-rxcfk_42b0c2c5-0b9f-4b0c-a782-6df4664999f9/extractor/0.log"
Sep 17 23:37:59.292175 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:37:59.292134    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-rhtfb_e4e101e1-5b0b-4b40-99cd-3c95101e61d4/s3-init/0.log"
Sep 17 23:37:59.313751 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:37:59.313726    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-rhtfb_e4e101e1-5b0b-4b40-99cd-3c95101e61d4/download-hf-model/0.log"
Sep 17 23:37:59.884389 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:37:59.884354    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-wk5sv/perf-node-gather-daemonset-7czn5"
Sep 17 23:38:03.742142 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:38:03.742058    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-76794cbd6c-zd4nf_88bdd5cd-67c8-4000-8dca-5c5875512672/migrator/0.log"
Sep 17 23:38:03.765634 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:38:03.765593    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-76794cbd6c-zd4nf_88bdd5cd-67c8-4000-8dca-5c5875512672/graceful-termination/0.log"
Sep 17 23:38:05.148804 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:38:05.148772    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-9j2wk_1c71bd0e-1761-4157-82d0-a33701aece84/kube-multus/0.log"
Sep 17 23:38:05.180610 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:38:05.180580    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-2gjg8_ea8e5a0a-14f1-4be6-84e1-33567e5d6032/kube-multus-additional-cni-plugins/0.log"
Sep 17 23:38:05.205549 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:38:05.205505    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-2gjg8_ea8e5a0a-14f1-4be6-84e1-33567e5d6032/egress-router-binary-copy/0.log"
Sep 17 23:38:05.234340 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:38:05.234317    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-2gjg8_ea8e5a0a-14f1-4be6-84e1-33567e5d6032/cni-plugins/0.log"
Sep 17 23:38:05.261444 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:38:05.261414    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-2gjg8_ea8e5a0a-14f1-4be6-84e1-33567e5d6032/bond-cni-plugin/0.log"
Sep 17 23:38:05.296785 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:38:05.296761    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-2gjg8_ea8e5a0a-14f1-4be6-84e1-33567e5d6032/routeoverride-cni/0.log"
Sep 17 23:38:05.334641 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:38:05.334614    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-2gjg8_ea8e5a0a-14f1-4be6-84e1-33567e5d6032/whereabouts-cni-bincopy/0.log"
Sep 17 23:38:05.370787 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:38:05.370759    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-2gjg8_ea8e5a0a-14f1-4be6-84e1-33567e5d6032/whereabouts-cni/0.log"
Sep 17 23:38:05.949911 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:38:05.949880    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-4bbdp_d672140f-ad59-4a68-9dde-80b62492af7f/network-metrics-daemon/0.log"
Sep 17 23:38:05.977765 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:38:05.977734    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-4bbdp_d672140f-ad59-4a68-9dde-80b62492af7f/kube-rbac-proxy/0.log"
Sep 17 23:38:07.588648 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:38:07.588617    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tcglq_123417a3-f884-4c2f-b7ac-837813412743/ovn-controller/0.log"
Sep 17 23:38:07.635491 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:38:07.635462    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tcglq_123417a3-f884-4c2f-b7ac-837813412743/ovn-acl-logging/0.log"
Sep 17 23:38:07.673674 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:38:07.673649    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tcglq_123417a3-f884-4c2f-b7ac-837813412743/kube-rbac-proxy-node/0.log"
Sep 17 23:38:07.706529 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:38:07.706495    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tcglq_123417a3-f884-4c2f-b7ac-837813412743/kube-rbac-proxy-ovn-metrics/0.log"
Sep 17 23:38:07.750802 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:38:07.750771    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tcglq_123417a3-f884-4c2f-b7ac-837813412743/northd/0.log"
Sep 17 23:38:07.784354 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:38:07.784326    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tcglq_123417a3-f884-4c2f-b7ac-837813412743/nbdb/0.log"
Sep 17 23:38:07.831852 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:38:07.831827    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tcglq_123417a3-f884-4c2f-b7ac-837813412743/sbdb/0.log"
Sep 17 23:38:07.950087 ip-10-0-142-4 kubenswrapper[2584]: I0917 23:38:07.949990    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tcglq_123417a3-f884-4c2f-b7ac-837813412743/ovnkube-controller/0.log"