Sep 17 17:31:45.230012 ip-10-0-134-96 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Sep 17 17:31:45.230022 ip-10-0-134-96 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Sep 17 17:31:45.230029 ip-10-0-134-96 systemd[1]: kubelet.service: Failed with result 'resources'.
Sep 17 17:31:45.230266 ip-10-0-134-96 systemd[1]: Failed to start Kubernetes Kubelet.
Sep 17 17:31:55.423025 ip-10-0-134-96 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Sep 17 17:31:55.423044 ip-10-0-134-96 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 429155f49ae34d2f815e191870837c3e --
Sep 17 17:34:03.221331 ip-10-0-134-96 systemd[1]: Starting Kubernetes Kubelet...
Sep 17 17:34:06.720064 ip-10-0-134-96 kubenswrapper[2574]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 17 17:34:06.720064 ip-10-0-134-96 kubenswrapper[2574]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 17 17:34:06.720064 ip-10-0-134-96 kubenswrapper[2574]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 17 17:34:06.720064 ip-10-0-134-96 kubenswrapper[2574]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 17 17:34:06.720064 ip-10-0-134-96 kubenswrapper[2574]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 17 17:34:06.720747 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:06.720149    2574 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 17 17:34:06.723796 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.723771    2574 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 17:34:06.723796 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.723795    2574 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 17:34:06.723796 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.723799    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 17:34:06.723796 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.723804    2574 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 17:34:06.723796 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.723807    2574 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 17:34:06.723940 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.723810    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 17:34:06.723940 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.723814    2574 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 17:34:06.723940 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.723818    2574 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 17:34:06.723940 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.723820    2574 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 17:34:06.723940 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.723823    2574 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 17:34:06.723940 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.723828    2574 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 17:34:06.723940 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.723832    2574 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 17:34:06.723940 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.723835    2574 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 17:34:06.723940 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.723838    2574 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 17:34:06.723940 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.723841    2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 17:34:06.723940 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.723844    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 17:34:06.723940 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.723847    2574 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 17:34:06.723940 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.723850    2574 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 17:34:06.723940 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.723853    2574 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 17:34:06.723940 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.723856    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 17:34:06.723940 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.723859    2574 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 17:34:06.723940 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.723862    2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 17:34:06.723940 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.723865    2574 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 17:34:06.723940 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.723868    2574 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 17:34:06.724450 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.723871    2574 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 17:34:06.724450 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.723875    2574 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 17:34:06.724450 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.723879    2574 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 17:34:06.724450 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.723882    2574 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 17:34:06.724450 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.723884    2574 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 17:34:06.724450 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.723887    2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 17:34:06.724450 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.723890    2574 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 17:34:06.724450 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.723893    2574 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 17:34:06.724450 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.723895    2574 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 17:34:06.724450 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.723898    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 17:34:06.724450 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.723900    2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 17:34:06.724450 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.723903    2574 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 17:34:06.724450 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.723905    2574 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 17:34:06.724450 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.723908    2574 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 17:34:06.724450 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.723911    2574 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 17:34:06.724450 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.723915    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 17:34:06.724450 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.723917    2574 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 17:34:06.724450 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.723920    2574 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 17:34:06.724450 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.723923    2574 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 17:34:06.724450 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.723925    2574 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 17:34:06.724932 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.723928    2574 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 17:34:06.724932 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.723931    2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 17:34:06.724932 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.723933    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 17:34:06.724932 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.723936    2574 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 17:34:06.724932 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.723938    2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 17:34:06.724932 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.723941    2574 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 17:34:06.724932 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.723944    2574 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 17:34:06.724932 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.723946    2574 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 17:34:06.724932 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.723949    2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 17:34:06.724932 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.723952    2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 17:34:06.724932 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.723955    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 17:34:06.724932 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.723957    2574 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 17:34:06.724932 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.723960    2574 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 17:34:06.724932 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.723963    2574 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 17:34:06.724932 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.723965    2574 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 17:34:06.724932 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.723967    2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 17:34:06.724932 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.723971    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 17:34:06.724932 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.723974    2574 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 17:34:06.724932 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.723976    2574 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 17:34:06.725431 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.723979    2574 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 17:34:06.725431 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.723981    2574 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 17:34:06.725431 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.723984    2574 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 17:34:06.725431 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.723986    2574 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 17:34:06.725431 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.723989    2574 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 17:34:06.725431 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.723991    2574 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 17:34:06.725431 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.723994    2574 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 17:34:06.725431 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.723998    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 17:34:06.725431 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.724001    2574 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 17:34:06.725431 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.724003    2574 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 17:34:06.725431 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.724006    2574 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 17:34:06.725431 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.724009    2574 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 17:34:06.725431 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.724011    2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 17:34:06.725431 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.724014    2574 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 17:34:06.725431 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.724017    2574 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 17:34:06.725431 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.724019    2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 17:34:06.725431 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.724022    2574 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 17:34:06.725431 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.724024    2574 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 17:34:06.725431 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.724027    2574 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 17:34:06.725431 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.724029    2574 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 17:34:06.725912 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.724031    2574 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 17:34:06.725912 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.724034    2574 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 17:34:06.725912 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.724037    2574 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 17:34:06.725912 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.724052    2574 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 17:34:06.725912 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.724055    2574 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 17:34:06.725912 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.724057    2574 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 17:34:06.725912 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.724060    2574 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 17:34:06.725912 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.724063    2574 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 17:34:06.725912 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.724066    2574 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 17:34:06.725912 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.724070    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 17:34:06.725912 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.724073    2574 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 17:34:06.725912 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.724076    2574 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 17:34:06.725912 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.724079    2574 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 17:34:06.725912 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.724082    2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 17:34:06.725912 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.724084    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 17:34:06.725912 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.724087    2574 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 17:34:06.725912 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.724749    2574 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 17:34:06.725912 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.724756    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 17:34:06.725912 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.724758    2574 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 17:34:06.726393 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.724763    2574 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 17:34:06.726393 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.724766    2574 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 17:34:06.726393 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.724768    2574 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 17:34:06.726393 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.724771    2574 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 17:34:06.726393 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.724774    2574 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 17:34:06.726393 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.724776    2574 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 17:34:06.726393 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.724780    2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 17:34:06.726393 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.724783    2574 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 17:34:06.726393 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.724786    2574 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 17:34:06.726393 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.724788    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 17:34:06.726393 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.724791    2574 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 17:34:06.726393 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.724794    2574 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 17:34:06.726393 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.724797    2574 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 17:34:06.726393 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.724799    2574 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 17:34:06.726393 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.724802    2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 17:34:06.726393 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.724804    2574 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 17:34:06.726393 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.724807    2574 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 17:34:06.726393 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.724810    2574 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 17:34:06.726393 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.724812    2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 17:34:06.726901 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.724816    2574 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 17:34:06.726901 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.724820    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 17:34:06.726901 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.724822    2574 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 17:34:06.726901 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.724825    2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 17:34:06.726901 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.724827    2574 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 17:34:06.726901 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.724830    2574 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 17:34:06.726901 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.724833    2574 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 17:34:06.726901 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.724835    2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 17:34:06.726901 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.724838    2574 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 17:34:06.726901 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.724840    2574 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 17:34:06.726901 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.724843    2574 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 17:34:06.726901 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.724845    2574 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 17:34:06.726901 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.724847    2574 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 17:34:06.726901 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.724851    2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 17:34:06.726901 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.724853    2574 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 17:34:06.726901 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.724856    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 17:34:06.726901 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.724859    2574 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 17:34:06.726901 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.724863    2574 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 17:34:06.726901 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.724867    2574 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 17:34:06.727394 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.724870    2574 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 17:34:06.727394 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.724876    2574 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 17:34:06.727394 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.724879    2574 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 17:34:06.727394 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.724881    2574 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 17:34:06.727394 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.724884    2574 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 17:34:06.727394 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.724886    2574 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 17:34:06.727394 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.724889    2574 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 17:34:06.727394 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.724891    2574 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 17:34:06.727394 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.724894    2574 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 17:34:06.727394 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.724897    2574 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 17:34:06.727394 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.724899    2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 17:34:06.727394 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.724902    2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 17:34:06.727394 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.724905    2574 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 17:34:06.727394 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.724907    2574 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 17:34:06.727394 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.724910    2574 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 17:34:06.727394 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.724912    2574 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 17:34:06.727394 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.724915    2574 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 17:34:06.727394 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.724918    2574 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 17:34:06.727394 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.724921    2574 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 17:34:06.727394 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.724924    2574 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 17:34:06.727893 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.724926    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 17:34:06.727893 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.724929    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 17:34:06.727893 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.724932    2574 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 17:34:06.727893 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.724934    2574 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 17:34:06.727893 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.724937    2574 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 17:34:06.727893 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.724940    2574 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 17:34:06.727893 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.724943    2574 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 17:34:06.727893 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.724946    2574 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 17:34:06.727893 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.724948    2574 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 17:34:06.727893 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.724951    2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 17:34:06.727893 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.724954    2574 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 17:34:06.727893 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.724957    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 17:34:06.727893 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.724961    2574 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 17:34:06.727893 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.724965    2574 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 17:34:06.727893 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.724968    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 17:34:06.727893 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.724971    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 17:34:06.727893 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.724974    2574 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 17:34:06.727893 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.724976    2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 17:34:06.727893 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.724979    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 17:34:06.728384 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.724982    2574 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 17:34:06.728384 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.724984    2574 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 17:34:06.728384 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.724987    2574 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 17:34:06.728384 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.724989    2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 17:34:06.728384 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.724992    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 17:34:06.728384 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.724994    2574 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 17:34:06.728384 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.724997    2574 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 17:34:06.728384 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.724999    2574 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 17:34:06.728384 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.725002    2574 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 17:34:06.728384 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.725004    2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 17:34:06.728384 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.725007    2574 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 17:34:06.728384 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.725010    2574 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 17:34:06.728384 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.725013    2574 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 17:34:06.728384 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.725015    2574 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 17:34:06.728384 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.725018    2574 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 17:34:06.728384 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.725020    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 17:34:06.728384 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.725023    2574 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 17:34:06.728384 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.725026    2574 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 17:34:06.728384 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.725028    2574 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 17:34:06.728384 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.725392    2574 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 17:34:06.728923 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.725397    2574 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 17:34:06.728923 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.725400    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 17:34:06.728923 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.725402    2574 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 17:34:06.728923 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.725405    2574 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 17:34:06.728923 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.725408    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 17:34:06.728923 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.725411    2574 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 17:34:06.728923 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.725413    2574 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 17:34:06.728923 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.725417    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 17:34:06.728923 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.725419    2574 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 17:34:06.728923 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.725422    2574 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 17:34:06.728923 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.725425    2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 17:34:06.728923 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.725427    2574 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 17:34:06.728923 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.725430    2574 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 17:34:06.728923 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.725433    2574 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 17:34:06.728923 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.725435    2574 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 17:34:06.728923 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.725438    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 17:34:06.728923 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.725441    2574 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 17:34:06.728923 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.725443    2574 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 17:34:06.728923 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.725446    2574 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 17:34:06.728923 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.725449    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 17:34:06.729429 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.725451    2574 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 17:34:06.729429 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.725454    2574 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 17:34:06.729429 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.725456    2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 17:34:06.729429 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.725459    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 17:34:06.729429 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.725462    2574 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 17:34:06.729429 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.725464    2574 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 17:34:06.729429 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.725467    2574 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 17:34:06.729429 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.725470    2574 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 17:34:06.729429 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.725472    2574 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 17:34:06.729429 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.725475    2574 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 17:34:06.729429 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.725477    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 17:34:06.729429 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.725482    2574 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 17:34:06.729429 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.725484    2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 17:34:06.729429 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.725487    2574 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 17:34:06.729429 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.725491    2574 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 17:34:06.729429 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.725494    2574 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 17:34:06.729429 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.725497    2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 17:34:06.729429 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.725500    2574 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 17:34:06.729429 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.725502    2574 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 17:34:06.729429 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.725505    2574 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 17:34:06.729914 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.725508    2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 17:34:06.729914 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.725511    2574 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 17:34:06.729914 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.725513    2574 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 17:34:06.729914 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.725516    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 17:34:06.729914 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.725518    2574 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 17:34:06.729914 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.725521    2574 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 17:34:06.729914 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.725524    2574 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 17:34:06.729914 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.725527    2574 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 17:34:06.729914 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.725530    2574 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 17:34:06.729914 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.725532    2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 17:34:06.729914 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.725536    2574 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 17:34:06.729914 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.725538    2574 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 17:34:06.729914 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.725541    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 17:34:06.729914 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.725545    2574 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 17:34:06.729914 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.725548    2574 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 17:34:06.729914 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.725551    2574 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 17:34:06.729914 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.725554    2574 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 17:34:06.729914 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.725557    2574 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 17:34:06.729914 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.725560    2574 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 17:34:06.730383 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.725562    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 17:34:06.730383 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.725565    2574 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 17:34:06.730383 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.725567    2574 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 17:34:06.730383 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.725570    2574 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 17:34:06.730383 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.725573    2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 17:34:06.730383 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.725576    2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 17:34:06.730383 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.725579    2574 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 17:34:06.730383 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.725582    2574 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 17:34:06.730383 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.725585    2574 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 17:34:06.730383 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.725587    2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 17:34:06.730383 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.725590    2574 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 17:34:06.730383 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.725592    2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 17:34:06.730383 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.725595    2574 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 17:34:06.730383 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.725597    2574 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 17:34:06.730383 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.725601    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 17:34:06.730383 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.725603    2574 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 17:34:06.730383 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.725605    2574 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 17:34:06.730383 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.725608    2574 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 17:34:06.730383 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.725611    2574 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 17:34:06.730383 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.725616    2574 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 17:34:06.730871 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.725620    2574 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 17:34:06.730871 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.725623    2574 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 17:34:06.730871 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.725626    2574 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 17:34:06.730871 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.725629    2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 17:34:06.730871 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.725631    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 17:34:06.730871 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.725634    2574 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 17:34:06.730871 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.725637    2574 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 17:34:06.730871 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.725639    2574 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 17:34:06.730871 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.725642    2574 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 17:34:06.730871 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.725645    2574 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 17:34:06.730871 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.725647    2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 17:34:06.730871 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.725649    2574 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 17:34:06.730871 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.725652    2574 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 17:34:06.730871 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.725655    2574 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 17:34:06.730871 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.725657    2574 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 17:34:06.730871 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.725660    2574 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 17:34:06.730871 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.725663    2574 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 17:34:06.730871 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.725666    2574 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 17:34:06.730871 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.725669    2574 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 17:34:06.730871 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:06.725744    2574 flags.go:64] FLAG: --address="0.0.0.0"
Sep 17 17:34:06.731374 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:06.725774    2574 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 17 17:34:06.731374 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:06.725782    2574 flags.go:64] FLAG: --anonymous-auth="true"
Sep 17 17:34:06.731374 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:06.725786    2574 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 17 17:34:06.731374 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:06.725790    2574 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 17 17:34:06.731374 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:06.725793    2574 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 17 17:34:06.731374 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:06.725797    2574 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 17 17:34:06.731374 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:06.725802    2574 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 17 17:34:06.731374 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:06.725806    2574 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 17 17:34:06.731374 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:06.725809    2574 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 17 17:34:06.731374 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:06.725812    2574 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 17 17:34:06.731374 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:06.725815    2574 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 17 17:34:06.731374 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:06.725818    2574 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 17 17:34:06.731374 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:06.725823    2574 flags.go:64] FLAG: --cgroup-root=""
Sep 17 17:34:06.731374 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:06.725826    2574 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 17 17:34:06.731374 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:06.725829    2574 flags.go:64] FLAG: --client-ca-file=""
Sep 17 17:34:06.731374 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:06.725832    2574 flags.go:64] FLAG: --cloud-provider="external"
Sep 17 17:34:06.731374 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:06.725835    2574 flags.go:64] FLAG: --cluster-dns="[]"
Sep 17 17:34:06.731374 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:06.725839    2574 flags.go:64] FLAG: --cluster-domain=""
Sep 17 17:34:06.731374 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:06.725842    2574 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 17 17:34:06.731374 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:06.725845    2574 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 17 17:34:06.731374 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:06.725848    2574 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 17 17:34:06.731374 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:06.725851    2574 flags.go:64] FLAG: --container-log-max-files="5"
Sep 17 17:34:06.731374 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:06.725855    2574 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 17 17:34:06.731374 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:06.725858    2574 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 17 17:34:06.731947 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:06.725861    2574 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 17 17:34:06.731947 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:06.725864    2574 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 17 17:34:06.731947 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:06.725867    2574 flags.go:64] FLAG: --contention-profiling="false"
Sep 17 17:34:06.731947 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:06.725870    2574 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 17 17:34:06.731947 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:06.725874    2574 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 17 17:34:06.731947 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:06.725877    2574 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 17 17:34:06.731947 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:06.725880    2574 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 17 17:34:06.731947 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:06.725884    2574 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 17 17:34:06.731947 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:06.725887    2574 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 17 17:34:06.731947 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:06.725890    2574 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 17 17:34:06.731947 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:06.725893    2574 flags.go:64] FLAG: --enable-load-reader="false"
Sep 17 17:34:06.731947 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:06.725896    2574 flags.go:64] FLAG: --enable-server="true"
Sep 17 17:34:06.731947 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:06.725899    2574 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 17 17:34:06.731947 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:06.725904    2574 flags.go:64] FLAG: --event-burst="100"
Sep 17 17:34:06.731947 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:06.725907    2574 flags.go:64] FLAG: --event-qps="50"
Sep 17 17:34:06.731947 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:06.725909    2574 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 17 17:34:06.731947 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:06.725914    2574 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 17 17:34:06.731947 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:06.725917    2574 flags.go:64] FLAG: --eviction-hard=""
Sep 17 17:34:06.731947 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:06.725920    2574 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 17 17:34:06.731947 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:06.725923    2574 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 17 17:34:06.731947 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:06.725928    2574 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 17 17:34:06.731947 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:06.725931    2574 flags.go:64] FLAG: --eviction-soft=""
Sep 17 17:34:06.731947 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:06.725934    2574 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 17 17:34:06.731947 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:06.725936    2574 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 17 17:34:06.731947 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:06.725939    2574 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 17 17:34:06.732645 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:06.725942    2574 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 17 17:34:06.732645 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:06.725945    2574 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 17 17:34:06.732645 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:06.725948    2574 flags.go:64] FLAG: --fail-swap-on="true"
Sep 17 17:34:06.732645 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:06.725972    2574 flags.go:64] FLAG: --feature-gates=""
Sep 17 17:34:06.732645 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:06.725977    2574 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 17 17:34:06.732645 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:06.725980    2574 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 17 17:34:06.732645 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:06.725984    2574 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 17 17:34:06.732645 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:06.725987    2574 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 17 17:34:06.732645 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:06.725990    2574 flags.go:64] FLAG: --healthz-port="10248"
Sep 17 17:34:06.732645 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:06.725993    2574 flags.go:64] FLAG: --help="false"
Sep 17 17:34:06.732645 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:06.725996    2574 flags.go:64] FLAG: --hostname-override="ip-10-0-134-96.ec2.internal"
Sep 17 17:34:06.732645 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:06.726001    2574 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 17 17:34:06.732645 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:06.726003    2574 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 17 17:34:06.732645 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:06.726006    2574 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 17 17:34:06.732645 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:06.726010    2574 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 17 17:34:06.732645 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:06.726013    2574 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 17 17:34:06.732645 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:06.726016    2574 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 17 17:34:06.732645 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:06.726019    2574 flags.go:64] FLAG: --image-service-endpoint=""
Sep 17 17:34:06.732645 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:06.726021    2574 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 17 17:34:06.732645 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:06.726024    2574 flags.go:64] FLAG: --kube-api-burst="100"
Sep 17 17:34:06.732645 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:06.726027    2574 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 17 17:34:06.732645 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:06.726030    2574 flags.go:64] FLAG: --kube-api-qps="50"
Sep 17 17:34:06.732645 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:06.726033    2574 flags.go:64] FLAG: --kube-reserved=""
Sep 17 17:34:06.732645 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:06.726036    2574 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 17 17:34:06.733274 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:06.726054    2574 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 17 17:34:06.733274 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:06.726057    2574 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 17 17:34:06.733274 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:06.726060    2574 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 17 17:34:06.733274 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:06.726067    2574 flags.go:64] FLAG: --lock-file=""
Sep 17 17:34:06.733274 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:06.726071    2574 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 17 17:34:06.733274 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:06.726074    2574 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 17 17:34:06.733274 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:06.726077    2574 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 17 17:34:06.733274 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:06.726082    2574 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 17 17:34:06.733274 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:06.726085    2574 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 17 17:34:06.733274 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:06.726087    2574 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 17 17:34:06.733274 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:06.726090    2574 flags.go:64] FLAG: --logging-format="text"
Sep 17 17:34:06.733274 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:06.726093    2574 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 17 17:34:06.733274 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:06.726096    2574 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 17 17:34:06.733274 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:06.726099    2574 flags.go:64] FLAG: --manifest-url=""
Sep 17 17:34:06.733274 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:06.726102    2574 flags.go:64] FLAG: --manifest-url-header=""
Sep 17 17:34:06.733274 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:06.726107    2574 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 17 17:34:06.733274 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:06.726109    2574 flags.go:64] FLAG: --max-open-files="1000000"
Sep 17 17:34:06.733274 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:06.726113    2574 flags.go:64] FLAG: --max-pods="110"
Sep 17 17:34:06.733274 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:06.726116    2574 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 17 17:34:06.733274 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:06.726120    2574 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 17 17:34:06.733274 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:06.726122    2574 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 17 17:34:06.733274 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:06.726126    2574 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 17 17:34:06.733274 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:06.726128    2574 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 17 17:34:06.733274 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:06.726131    2574 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 17 17:34:06.733274 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:06.726134    2574 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 17 17:34:06.733887 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:06.726141    2574 flags.go:64] FLAG: --node-status-max-images="50"
Sep 17 17:34:06.733887 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:06.726144    2574 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 17 17:34:06.733887 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:06.726147    2574 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 17 17:34:06.733887 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:06.726150    2574 flags.go:64] FLAG: --pod-cidr=""
Sep 17 17:34:06.733887 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:06.726153    2574 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a6abe77af61c5f2e3985a83e7d43538aefc38b1b0444127da14ee0e8cf8f76fc"
Sep 17 17:34:06.733887 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:06.726158    2574 flags.go:64] FLAG: --pod-manifest-path=""
Sep 17 17:34:06.733887 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:06.726161    2574 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 17 17:34:06.733887 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:06.726164    2574 flags.go:64] FLAG: --pods-per-core="0"
Sep 17 17:34:06.733887 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:06.726168    2574 flags.go:64] FLAG: --port="10250"
Sep 17 17:34:06.733887 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:06.726171    2574 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 17 17:34:06.733887 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:06.726175    2574 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-05bfadea8c8af15d4"
Sep 17 17:34:06.733887 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:06.726179    2574 flags.go:64] FLAG: --qos-reserved=""
Sep 17 17:34:06.733887 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:06.726182    2574 flags.go:64] FLAG: --read-only-port="10255"
Sep 17 17:34:06.733887 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:06.726185    2574 flags.go:64] FLAG: --register-node="true"
Sep 17 17:34:06.733887 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:06.726188    2574 flags.go:64] FLAG: --register-with-taints=""
Sep 17 17:34:06.733887 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:06.726191    2574 flags.go:64] FLAG: --registry-burst="10"
Sep 17 17:34:06.733887 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:06.726194    2574 flags.go:64] FLAG: --registry-qps="5"
Sep 17 17:34:06.733887 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:06.726197    2574 flags.go:64] FLAG: --reserved-cpus=""
Sep 17 17:34:06.733887 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:06.726200    2574 flags.go:64] FLAG: --reserved-memory=""
Sep 17 17:34:06.733887 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:06.726203    2574 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 17 17:34:06.733887 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:06.726206    2574 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 17 17:34:06.733887 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:06.726209    2574 flags.go:64] FLAG: --rotate-certificates="false"
Sep 17 17:34:06.733887 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:06.726212    2574 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 17 17:34:06.733887 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:06.726215    2574 flags.go:64] FLAG: --runonce="false"
Sep 17 17:34:06.733887 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:06.726218    2574 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 17 17:34:06.734555 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:06.726221    2574 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 17 17:34:06.734555 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:06.726224    2574 flags.go:64] FLAG: --seccomp-default="false"
Sep 17 17:34:06.734555 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:06.726227    2574 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 17 17:34:06.734555 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:06.726230    2574 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 17 17:34:06.734555 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:06.726233    2574 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 17 17:34:06.734555 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:06.726236    2574 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 17 17:34:06.734555 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:06.726239    2574 flags.go:64] FLAG: --storage-driver-password="root"
Sep 17 17:34:06.734555 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:06.726241    2574 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 17 17:34:06.734555 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:06.726244    2574 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 17 17:34:06.734555 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:06.726247    2574 flags.go:64] FLAG: --storage-driver-user="root"
Sep 17 17:34:06.734555 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:06.726250    2574 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 17 17:34:06.734555 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:06.726253    2574 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 17 17:34:06.734555 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:06.726255    2574 flags.go:64] FLAG: --system-cgroups=""
Sep 17 17:34:06.734555 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:06.726258    2574 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 17 17:34:06.734555 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:06.726263    2574 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 17 17:34:06.734555 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:06.726266    2574 flags.go:64] FLAG: --tls-cert-file=""
Sep 17 17:34:06.734555 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:06.726269    2574 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 17 17:34:06.734555 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:06.726274    2574 flags.go:64] FLAG: --tls-min-version=""
Sep 17 17:34:06.734555 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:06.726277    2574 flags.go:64] FLAG: --tls-private-key-file=""
Sep 17 17:34:06.734555 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:06.726280    2574 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 17 17:34:06.734555 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:06.726282    2574 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 17 17:34:06.734555 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:06.726285    2574 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 17 17:34:06.734555 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:06.726288    2574 flags.go:64] FLAG: --v="2"
Sep 17 17:34:06.734555 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:06.726292    2574 flags.go:64] FLAG: --version="false"
Sep 17 17:34:06.734555 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:06.726295    2574 flags.go:64] FLAG: --vmodule=""
Sep 17 17:34:06.735174 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:06.726299    2574 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 17 17:34:06.735174 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:06.726302    2574 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 17 17:34:06.735174 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.726413    2574 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 17:34:06.735174 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.726417    2574 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 17:34:06.735174 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.726420    2574 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 17:34:06.735174 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.726424    2574 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 17:34:06.735174 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.726428    2574 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 17:34:06.735174 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.726431    2574 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 17:34:06.735174 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.726434    2574 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 17:34:06.735174 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.726437    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 17:34:06.735174 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.726440    2574 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 17:34:06.735174 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.726443    2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 17:34:06.735174 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.726447    2574 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 17:34:06.735174 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.726450    2574 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 17:34:06.735174 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.726453    2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 17:34:06.735174 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.726455    2574 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 17:34:06.735174 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.726458    2574 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 17:34:06.735174 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.726461    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 17:34:06.735174 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.726464    2574 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 17:34:06.735174 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.726466    2574 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 17:34:06.735713 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.726469    2574 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 17:34:06.735713 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.726471    2574 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 17:34:06.735713 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.726474    2574 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 17:34:06.735713 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.726476    2574 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 17:34:06.735713 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.726481    2574 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 17:34:06.735713 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.726484    2574 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 17:34:06.735713 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.726486    2574 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 17:34:06.735713 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.726489    2574 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 17:34:06.735713 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.726492    2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 17:34:06.735713 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.726494    2574 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 17:34:06.735713 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.726497    2574 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 17:34:06.735713 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.726499    2574 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 17:34:06.735713 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.726502    2574 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 17:34:06.735713 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.726505    2574 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 17:34:06.735713 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.726507    2574 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 17:34:06.735713 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.726510    2574 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 17:34:06.735713 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.726512    2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 17:34:06.735713 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.726522    2574 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 17:34:06.735713 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.726525    2574 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 17:34:06.735713 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.726527    2574 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 17:34:06.736294 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.726530    2574 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 17:34:06.736294 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.726534    2574 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 17:34:06.736294 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.726536    2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 17:34:06.736294 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.726539    2574 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 17:34:06.736294 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.726542    2574 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 17:34:06.736294 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.726544    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 17:34:06.736294 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.726546    2574 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 17:34:06.736294 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.726549    2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 17:34:06.736294 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.726552    2574 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 17:34:06.736294 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.726554    2574 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 17:34:06.736294 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.726557    2574 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 17:34:06.736294 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.726559    2574 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 17:34:06.736294 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.726562    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 17:34:06.736294 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.726564    2574 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 17:34:06.736294 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.726568    2574 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 17:34:06.736294 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.726571    2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 17:34:06.736294 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.726575    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 17:34:06.736294 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.726578    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 17:34:06.736294 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.726581    2574 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 17:34:06.736956 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.726584    2574 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 17:34:06.736956 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.726586    2574 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 17:34:06.736956 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.726589    2574 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 17:34:06.736956 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.726592    2574 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 17:34:06.736956 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.726594    2574 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 17:34:06.736956 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.726597    2574 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 17:34:06.736956 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.726600    2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 17:34:06.736956 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.726602    2574 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 17:34:06.736956 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.726605    2574 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 17:34:06.736956 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.726607    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 17:34:06.736956 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.726610    2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 17:34:06.736956 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.726612    2574 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 17:34:06.736956 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.726615    2574 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 17:34:06.736956 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.726618    2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 17:34:06.736956 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.726622    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 17:34:06.736956 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.726624    2574 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 17:34:06.736956 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.726627    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 17:34:06.736956 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.726630    2574 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 17:34:06.736956 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.726632    2574 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 17:34:06.736956 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.726635    2574 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 17:34:06.737482 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.726637    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 17:34:06.737482 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.726640    2574 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 17:34:06.737482 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.726643    2574 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 17:34:06.737482 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.726646    2574 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 17:34:06.737482 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.726648    2574 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 17:34:06.737482 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.726651    2574 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 17:34:06.737482 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.726653    2574 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 17:34:06.737482 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.726656    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 17:34:06.737482 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.726660    2574 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 17:34:06.737482 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.726665    2574 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 17:34:06.737482 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.726668    2574 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 17:34:06.737482 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.726672    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 17:34:06.737482 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.726675    2574 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 17:34:06.737482 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.726678    2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 17:34:06.737482 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.726680    2574 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 17:34:06.737482 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.726683    2574 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 17:34:06.737482 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.726685    2574 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 17:34:06.737482 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.726688    2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 17:34:06.737482 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.726690    2574 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 17:34:06.737482 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.726693    2574 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 17:34:06.738001 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.726696    2574 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 17:34:06.738001 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.726698    2574 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 17:34:06.738001 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:06.726703    2574 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 17 17:34:06.738001 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:06.733402    2574 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 17 17:34:06.738001 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:06.733416    2574 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 17 17:34:06.738001 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:06.733435    2574 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 17 17:34:06.738001 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:06.733441    2574 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 17 17:34:06.738001 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.733505    2574 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 17:34:06.738001 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.733510    2574 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 17:34:06.738001 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.733514    2574 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 17:34:06.738001 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.733517    2574 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 17:34:06.738001 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.733520    2574 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 17:34:06.738001 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.733523    2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 17:34:06.738001 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.733526    2574 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 17:34:06.738001 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.733528    2574 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 17:34:06.738467 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.733531    2574 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 17:34:06.738467 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.733534    2574 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 17:34:06.738467 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.733536    2574 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 17:34:06.738467 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.733539    2574 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 17:34:06.738467 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.733541    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 17:34:06.738467 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.733544    2574 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 17:34:06.738467 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.733547    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 17:34:06.738467 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.733550    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 17:34:06.738467 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.733552    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 17:34:06.738467 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.733555    2574 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 17:34:06.738467 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.733557    2574 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 17:34:06.738467 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.733560    2574 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 17:34:06.738467 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.733562    2574 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 17:34:06.738467 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.733565    2574 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 17:34:06.738467 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.733568    2574 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 17:34:06.738467 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.733570    2574 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 17:34:06.738467 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.733573    2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 17:34:06.738467 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.733576    2574 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 17:34:06.738467 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.733578    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 17:34:06.738467 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.733581    2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 17:34:06.738979 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.733584    2574 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 17:34:06.738979 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.733588    2574 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 17:34:06.738979 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.733590    2574 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 17:34:06.738979 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.733593    2574 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 17:34:06.738979 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.733595    2574 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 17:34:06.738979 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.733598    2574 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 17:34:06.738979 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.733601    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 17:34:06.738979 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.733604    2574 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 17:34:06.738979 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.733606    2574 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 17:34:06.738979 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.733609    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 17:34:06.738979 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.733611    2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 17:34:06.738979 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.733614    2574 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 17:34:06.738979 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.733617    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 17:34:06.738979 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.733619    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 17:34:06.738979 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.733622    2574 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 17:34:06.738979 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.733624    2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 17:34:06.738979 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.733627    2574 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 17:34:06.738979 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.733629    2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 17:34:06.738979 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.733632    2574 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 17:34:06.738979 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.733634    2574 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 17:34:06.739484 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.733637    2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 17:34:06.739484 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.733639    2574 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 17:34:06.739484 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.733642    2574 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 17:34:06.739484 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.733645    2574 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 17:34:06.739484 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.733647    2574 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 17:34:06.739484 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.733650    2574 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 17:34:06.739484 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.733653    2574 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 17:34:06.739484 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.733656    2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 17:34:06.739484 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.733659    2574 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 17:34:06.739484 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.733663    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 17:34:06.739484 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.733666    2574 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 17:34:06.739484 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.733669    2574 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 17:34:06.739484 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.733671    2574 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 17:34:06.739484 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.733675    2574 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 17:34:06.739484 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.733678    2574 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 17:34:06.739484 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.733681    2574 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 17:34:06.739484 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.733684    2574 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 17:34:06.739484 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.733686    2574 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 17:34:06.739484 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.733689    2574 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 17:34:06.739982 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.733691    2574 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 17:34:06.739982 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.733694    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 17:34:06.739982 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.733696    2574 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 17:34:06.739982 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.733699    2574 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 17:34:06.739982 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.733701    2574 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 17:34:06.739982 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.733704    2574 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 17:34:06.739982 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.733707    2574 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 17:34:06.739982 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.733710    2574 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 17:34:06.739982 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.733712    2574 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 17:34:06.739982 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.733715    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 17:34:06.739982 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.733717    2574 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 17:34:06.739982 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.733720    2574 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 17:34:06.739982 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.733723    2574 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 17:34:06.739982 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.733727    2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 17:34:06.739982 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.733730    2574 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 17:34:06.739982 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.733734    2574 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 17:34:06.739982 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.733737    2574 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 17:34:06.739982 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.733740    2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 17:34:06.739982 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.733743    2574 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 17:34:06.740467 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.733746    2574 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 17:34:06.740467 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.733748    2574 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 17:34:06.740467 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.733751    2574 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 17:34:06.740467 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.733753    2574 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 17:34:06.740467 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.733756    2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 17:34:06.740467 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.733758    2574 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 17:34:06.740467 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.733761    2574 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 17:34:06.740467 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.733764    2574 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 17:34:06.740467 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.733766    2574 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 17:34:06.740467 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.733772    2574 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 17:34:06.740467 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.733774    2574 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 17:34:06.740467 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.733777    2574 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 17:34:06.740467 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.733780    2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 17:34:06.740467 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:06.733784    2574 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 17 17:34:06.740467 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.733889    2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 17:34:06.740847 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.733894    2574 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 17:34:06.740847 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.733897    2574 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 17:34:06.740847 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.733899    2574 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 17:34:06.740847 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.733902    2574 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 17:34:06.740847 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.733905    2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 17:34:06.740847 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.733908    2574 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 17:34:06.740847 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.733910    2574 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 17:34:06.740847 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.733913    2574 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 17:34:06.740847 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.733915    2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 17:34:06.740847 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.733918    2574 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 17:34:06.740847 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.733921    2574 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 17:34:06.740847 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.733923    2574 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 17:34:06.740847 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.733926    2574 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 17:34:06.740847 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.733930    2574 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 17:34:06.740847 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.733933    2574 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 17:34:06.740847 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.733936    2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 17:34:06.740847 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.733938    2574 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 17:34:06.740847 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.733941    2574 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 17:34:06.740847 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.733943    2574 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 17:34:06.741351 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.733946    2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 17:34:06.741351 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.733948    2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 17:34:06.741351 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.733951    2574 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 17:34:06.741351 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.733954    2574 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 17:34:06.741351 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.733956    2574 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 17:34:06.741351 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.733959    2574 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 17:34:06.741351 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.733962    2574 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 17:34:06.741351 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.733965    2574 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 17:34:06.741351 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.733968    2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 17:34:06.741351 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.733970    2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 17:34:06.741351 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.733973    2574 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 17:34:06.741351 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.733975    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 17:34:06.741351 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.733979    2574 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 17:34:06.741351 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.733982    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 17:34:06.741351 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.733985    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 17:34:06.741351 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.733988    2574 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 17:34:06.741351 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.733991    2574 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 17:34:06.741351 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.733994    2574 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 17:34:06.741351 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.733997    2574 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 17:34:06.741820 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.734000    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 17:34:06.741820 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.734002    2574 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 17:34:06.741820 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.734006    2574 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 17:34:06.741820 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.734009    2574 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 17:34:06.741820 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.734012    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 17:34:06.741820 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.734015    2574 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 17:34:06.741820 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.734018    2574 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 17:34:06.741820 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.734021    2574 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 17:34:06.741820 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.734024    2574 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 17:34:06.741820 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.734027    2574 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 17:34:06.741820 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.734030    2574 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 17:34:06.741820 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.734032    2574 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 17:34:06.741820 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.734035    2574 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 17:34:06.741820 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.734038    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 17:34:06.741820 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.734055    2574 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 17:34:06.741820 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.734060    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 17:34:06.741820 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.734063    2574 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 17:34:06.741820 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.734065    2574 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 17:34:06.741820 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.734068    2574 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 17:34:06.742364 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.734070    2574 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 17:34:06.742364 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.734073    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 17:34:06.742364 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.734076    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 17:34:06.742364 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.734079    2574 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 17:34:06.742364 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.734082    2574 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 17:34:06.742364 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.734084    2574 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 17:34:06.742364 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.734087    2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 17:34:06.742364 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.734090    2574 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 17:34:06.742364 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.734092    2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 17:34:06.742364 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.734095    2574 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 17:34:06.742364 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.734097    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 17:34:06.742364 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.734100    2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 17:34:06.742364 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.734103    2574 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 17:34:06.742364 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.734105    2574 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 17:34:06.742364 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.734108    2574 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 17:34:06.742364 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.734110    2574 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 17:34:06.742364 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.734112    2574 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 17:34:06.742364 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.734115    2574 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 17:34:06.742364 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.734117    2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 17:34:06.742364 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.734120    2574 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 17:34:06.742841 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.734123    2574 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 17:34:06.742841 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.734125    2574 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 17:34:06.742841 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.734128    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 17:34:06.742841 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.734130    2574 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 17:34:06.742841 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.734133    2574 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 17:34:06.742841 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.734135    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 17:34:06.742841 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.734138    2574 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 17:34:06.742841 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.734140    2574 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 17:34:06.742841 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.734143    2574 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 17:34:06.742841 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.734145    2574 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 17:34:06.742841 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.734148    2574 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 17:34:06.742841 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.734151    2574 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 17:34:06.742841 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.734153    2574 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 17:34:06.742841 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.734156    2574 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 17:34:06.742841 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.734158    2574 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 17:34:06.742841 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.734161    2574 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 17:34:06.742841 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.734164    2574 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 17:34:06.742841 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.734167    2574 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 17:34:06.742841 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.734170    2574 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 17:34:06.742841 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.734172    2574 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 17:34:06.743359 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:06.734175    2574 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 17:34:06.743359 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:06.734180    2574 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 17 17:34:06.743359 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:06.734274    2574 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 17 17:34:06.743359 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:06.737600    2574 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 17 17:34:06.743359 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:06.738576    2574 server.go:1019] "Starting client certificate rotation"
Sep 17 17:34:06.743359 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:06.738668    2574 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 17 17:34:06.743359 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:06.739399    2574 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 17 17:34:06.743359 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:06.741031    2574 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 17 17:34:06.743359 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:06.742075    2574 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 17 17:34:06.746334 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:06.746318    2574 log.go:25] "Validated CRI v1 runtime API"
Sep 17 17:34:06.748560 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:06.748540    2574 log.go:25] "Validated CRI v1 image API"
Sep 17 17:34:06.749727 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:06.749711    2574 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 17 17:34:06.751871 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:06.751843    2574 fs.go:135] Filesystem UUIDs: map[2b8df42d-c575-4cc8-b673-45e9f892237b:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 7ca06958-d8df-4978-b652-3cec0c4e5fd8:/dev/nvme0n1p4]
Sep 17 17:34:06.751946 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:06.751866    2574 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 17 17:34:06.756825 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:06.756710    2574 manager.go:216] Machine: {Timestamp:2026-09-17 17:34:06.755344017 +0000 UTC m=+1.944217715 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3103319 MemoryCapacity:33164480512 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec28385affde1b82b99d183adf5bb96c SystemUUID:ec28385a-ffde-1b82-b99d-183adf5bb96c BootID:429155f4-9ae3-4d2f-815e-191870837c3e Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582242304 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:cd:ff:2a:b3 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:cd:ff:2a:b3 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:4e:a7:f1:47:7b:ee Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164480512 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 17 17:34:06.756825 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:06.756815    2574 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 17 17:34:06.756972 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:06.756886    2574 manager.go:232] Version: {KernelVersion:5.14.0-570.141.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260914-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 17 17:34:06.758477 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:06.758457    2574 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 17 17:34:06.758596 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:06.758480    2574 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-134-96.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 17 17:34:06.759713 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:06.759703    2574 topology_manager.go:138] "Creating topology manager with none policy"
Sep 17 17:34:06.759752 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:06.759716    2574 container_manager_linux.go:309] "Creating device plugin manager"
Sep 17 17:34:06.759752 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:06.759723    2574 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 17 17:34:06.759752 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:06.759734    2574 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 17 17:34:06.759838 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:06.759794    2574 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 17 17:34:06.764394 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:06.764382    2574 state_mem.go:36] "Initialized new in-memory state store"
Sep 17 17:34:06.764507 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:06.764497    2574 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 17 17:34:06.764539 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:06.764531    2574 kubelet.go:486] "Attempting to sync node with API server"
Sep 17 17:34:06.764580 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:06.764544    2574 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 17 17:34:06.764580 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:06.764558    2574 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 17 17:34:06.764580 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:06.764567    2574 kubelet.go:398] "Adding apiserver pod source"
Sep 17 17:34:06.764580 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:06.764575    2574 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 17 17:34:06.764923 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:06.764909    2574 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 17 17:34:06.764983 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:06.764928    2574 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 17 17:34:06.767812 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:06.767792    2574 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.13-6.rhaos4.21.git6253162.el9" apiVersion="v1"
Sep 17 17:34:06.770305 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:06.770291    2574 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 17 17:34:06.770365 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:06.770321    2574 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 17 17:34:06.771366 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:06.771352    2574 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 17 17:34:06.771503 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:06.771480    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 17 17:34:06.771503 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:06.771502    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 17 17:34:06.771656 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:06.771510    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 17 17:34:06.771656 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:06.771518    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 17 17:34:06.771656 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:06.771529    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 17 17:34:06.771656 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:06.771536    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 17 17:34:06.771656 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:06.771554    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 17 17:34:06.771656 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:06.771561    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 17 17:34:06.771656 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:06.771571    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 17 17:34:06.771656 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:06.771579    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 17 17:34:06.771656 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:06.771592    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 17 17:34:06.771656 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:06.771600    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 17 17:34:06.771656 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:06.771633    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 17 17:34:06.771656 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:06.771643    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 17 17:34:06.775379 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:06.775363    2574 server.go:1268] "Started kubelet"
Sep 17 17:34:06.775493 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:06.775453    2574 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 17 17:34:06.775539 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:06.775461    2574 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 17 17:34:06.775539 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:06.775520    2574 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 17 17:34:06.776173 ip-10-0-134-96 systemd[1]: Started Kubernetes Kubelet.
Sep 17 17:34:06.776572 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:06.776544    2574 server.go:310] "Adding debug handlers to kubelet server"
Sep 17 17:34:06.779350 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:06.779334    2574 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 17 17:34:06.779488 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:06.779393    2574 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-134-96.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 17 17:34:06.779885 ip-10-0-134-96 kubenswrapper[2574]: E0917 17:34:06.779866    2574 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 17 17:34:06.780188 ip-10-0-134-96 kubenswrapper[2574]: E0917 17:34:06.780167    2574 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 17 17:34:06.780188 ip-10-0-134-96 kubenswrapper[2574]: E0917 17:34:06.780169    2574 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-134-96.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 17 17:34:06.784108 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:06.784095    2574 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 17 17:34:06.784176 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:06.784107    2574 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 17 17:34:06.784225 ip-10-0-134-96 kubenswrapper[2574]: E0917 17:34:06.784176    2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-96.ec2.internal\" not found"
Sep 17 17:34:06.784633 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:06.784618    2574 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 17 17:34:06.784633 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:06.784628    2574 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 17 17:34:06.784750 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:06.784634    2574 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 17 17:34:06.784750 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:06.784724    2574 reconstruct.go:97] "Volume reconstruction finished"
Sep 17 17:34:06.784750 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:06.784733    2574 reconciler.go:29] "Reconciler: start to sync state"
Sep 17 17:34:06.784920 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:06.784907    2574 factory.go:55] Registering systemd factory
Sep 17 17:34:06.784955 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:06.784924    2574 factory.go:223] Registration of the systemd container factory successfully
Sep 17 17:34:06.785131 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:06.785118    2574 factory.go:153] Registering CRI-O factory
Sep 17 17:34:06.785190 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:06.785134    2574 factory.go:223] Registration of the crio container factory successfully
Sep 17 17:34:06.785233 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:06.785197    2574 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 17 17:34:06.785233 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:06.785217    2574 factory.go:103] Registering Raw factory
Sep 17 17:34:06.785233 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:06.785232    2574 manager.go:1196] Started watching for new ooms in manager
Sep 17 17:34:06.786259 ip-10-0-134-96 kubenswrapper[2574]: E0917 17:34:06.786239    2574 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 17 17:34:06.786318 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:06.786266    2574 manager.go:349] Starting recovery of all containers
Sep 17 17:34:06.787361 ip-10-0-134-96 kubenswrapper[2574]: E0917 17:34:06.787338    2574 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-134-96.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 17 17:34:06.809157 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:06.809135    2574 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-9t2rx"
Sep 17 17:34:06.815014 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:06.814996    2574 manager.go:354] Recovery completed
Sep 17 17:34:06.819206 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:06.819195    2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 17:34:06.819638 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:06.819625    2574 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-9t2rx"
Sep 17 17:34:06.823587 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:06.823569    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-96.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 17:34:06.823640 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:06.823601    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-96.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 17:34:06.823640 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:06.823614    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-96.ec2.internal" event="NodeHasSufficientPID"
Sep 17 17:34:06.824116 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:06.824102    2574 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 17 17:34:06.824186 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:06.824116    2574 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 17 17:34:06.824186 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:06.824134    2574 state_mem.go:36] "Initialized new in-memory state store"
Sep 17 17:34:06.824186 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:06.824136    2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 17:34:06.824882 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:06.824867    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-96.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 17:34:06.824921 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:06.824899    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-96.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 17:34:06.824921 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:06.824914    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-96.ec2.internal" event="NodeHasSufficientPID"
Sep 17 17:34:06.826140 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:06.826129    2574 policy_none.go:49] "None policy: Start"
Sep 17 17:34:06.826184 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:06.826144    2574 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 17 17:34:06.826184 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:06.826155    2574 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 17 17:34:06.828131 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:06.828120    2574 policy_none.go:47] "Start"
Sep 17 17:34:06.881567 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:06.867562    2574 manager.go:337] "Starting Device Plugin manager"
Sep 17 17:34:06.881567 ip-10-0-134-96 kubenswrapper[2574]: E0917 17:34:06.867590    2574 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 17 17:34:06.881567 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:06.867599    2574 server.go:87] "Starting device plugin registration server"
Sep 17 17:34:06.881567 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:06.867789    2574 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 17 17:34:06.881567 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:06.867806    2574 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 17 17:34:06.881567 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:06.867893    2574 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 17 17:34:06.881567 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:06.867975    2574 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 17 17:34:06.881567 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:06.867983    2574 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 17 17:34:06.881567 ip-10-0-134-96 kubenswrapper[2574]: E0917 17:34:06.868428    2574 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 17 17:34:06.881567 ip-10-0-134-96 kubenswrapper[2574]: E0917 17:34:06.868467    2574 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-134-96.ec2.internal\" not found"
Sep 17 17:34:06.946555 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:06.946529    2574 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 17 17:34:06.947774 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:06.947749    2574 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 17 17:34:06.947861 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:06.947801    2574 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 17 17:34:06.947861 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:06.947824    2574 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 17 17:34:06.947944 ip-10-0-134-96 kubenswrapper[2574]: E0917 17:34:06.947867    2574 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 17 17:34:06.950210 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:06.950193    2574 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 17:34:06.968153 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:06.968139    2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 17:34:06.968965 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:06.968949    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-96.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 17:34:06.969064 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:06.968976    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-96.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 17:34:06.969064 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:06.968988    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-96.ec2.internal" event="NodeHasSufficientPID"
Sep 17 17:34:06.969064 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:06.969012    2574 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-134-96.ec2.internal"
Sep 17 17:34:06.978409 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:06.978373    2574 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-134-96.ec2.internal"
Sep 17 17:34:06.978409 ip-10-0-134-96 kubenswrapper[2574]: E0917 17:34:06.978393    2574 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-134-96.ec2.internal\": node \"ip-10-0-134-96.ec2.internal\" not found"
Sep 17 17:34:06.992189 ip-10-0-134-96 kubenswrapper[2574]: E0917 17:34:06.992170    2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-96.ec2.internal\" not found"
Sep 17 17:34:07.048397 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.048377    2574 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-96.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-134-96.ec2.internal"]
Sep 17 17:34:07.048463 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.048447    2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 17:34:07.049379 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.049363    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-96.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 17:34:07.049434 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.049392    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-96.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 17:34:07.049434 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.049404    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-96.ec2.internal" event="NodeHasSufficientPID"
Sep 17 17:34:07.051656 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.051644    2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 17:34:07.051784 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.051771    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-96.ec2.internal"
Sep 17 17:34:07.051820 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.051799    2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 17:34:07.052279 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.052263    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-96.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 17:34:07.052342 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.052293    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-96.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 17:34:07.052342 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.052307    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-96.ec2.internal" event="NodeHasSufficientPID"
Sep 17 17:34:07.052342 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.052317    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-96.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 17:34:07.052342 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.052333    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-96.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 17:34:07.052474 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.052354    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-96.ec2.internal" event="NodeHasSufficientPID"
Sep 17 17:34:07.054526 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.054513    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-96.ec2.internal"
Sep 17 17:34:07.054576 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.054535    2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 17:34:07.055229 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.055212    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-96.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 17:34:07.055314 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.055240    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-96.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 17:34:07.055314 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.055250    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-96.ec2.internal" event="NodeHasSufficientPID"
Sep 17 17:34:07.081660 ip-10-0-134-96 kubenswrapper[2574]: E0917 17:34:07.081638    2574 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-134-96.ec2.internal\" not found" node="ip-10-0-134-96.ec2.internal"
Sep 17 17:34:07.085973 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.085953    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5ecb72cbea487238ef07d1e2c3698290-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-96.ec2.internal\" (UID: \"5ecb72cbea487238ef07d1e2c3698290\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-96.ec2.internal"
Sep 17 17:34:07.086075 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.085986    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ee6fb7c1eb5fa5842b2aa3f6f4ac1869-config\") pod \"kube-apiserver-proxy-ip-10-0-134-96.ec2.internal\" (UID: \"ee6fb7c1eb5fa5842b2aa3f6f4ac1869\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-96.ec2.internal"
Sep 17 17:34:07.086075 ip-10-0-134-96 kubenswrapper[2574]: E0917 17:34:07.085994    2574 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-134-96.ec2.internal\" not found" node="ip-10-0-134-96.ec2.internal"
Sep 17 17:34:07.086075 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.086009    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/5ecb72cbea487238ef07d1e2c3698290-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-96.ec2.internal\" (UID: \"5ecb72cbea487238ef07d1e2c3698290\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-96.ec2.internal"
Sep 17 17:34:07.093053 ip-10-0-134-96 kubenswrapper[2574]: E0917 17:34:07.093030    2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-96.ec2.internal\" not found"
Sep 17 17:34:07.186847 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.186819    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/5ecb72cbea487238ef07d1e2c3698290-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-96.ec2.internal\" (UID: \"5ecb72cbea487238ef07d1e2c3698290\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-96.ec2.internal"
Sep 17 17:34:07.186847 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.186848    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5ecb72cbea487238ef07d1e2c3698290-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-96.ec2.internal\" (UID: \"5ecb72cbea487238ef07d1e2c3698290\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-96.ec2.internal"
Sep 17 17:34:07.186995 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.186864    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ee6fb7c1eb5fa5842b2aa3f6f4ac1869-config\") pod \"kube-apiserver-proxy-ip-10-0-134-96.ec2.internal\" (UID: \"ee6fb7c1eb5fa5842b2aa3f6f4ac1869\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-96.ec2.internal"
Sep 17 17:34:07.186995 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.186906    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ee6fb7c1eb5fa5842b2aa3f6f4ac1869-config\") pod \"kube-apiserver-proxy-ip-10-0-134-96.ec2.internal\" (UID: \"ee6fb7c1eb5fa5842b2aa3f6f4ac1869\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-96.ec2.internal"
Sep 17 17:34:07.186995 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.186932    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/5ecb72cbea487238ef07d1e2c3698290-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-96.ec2.internal\" (UID: \"5ecb72cbea487238ef07d1e2c3698290\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-96.ec2.internal"
Sep 17 17:34:07.186995 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.186933    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5ecb72cbea487238ef07d1e2c3698290-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-96.ec2.internal\" (UID: \"5ecb72cbea487238ef07d1e2c3698290\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-96.ec2.internal"
Sep 17 17:34:07.193857 ip-10-0-134-96 kubenswrapper[2574]: E0917 17:34:07.193842    2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-96.ec2.internal\" not found"
Sep 17 17:34:07.294135 ip-10-0-134-96 kubenswrapper[2574]: E0917 17:34:07.294078    2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-96.ec2.internal\" not found"
Sep 17 17:34:07.385309 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.385285    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-96.ec2.internal"
Sep 17 17:34:07.389106 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.389090    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-96.ec2.internal"
Sep 17 17:34:07.395278 ip-10-0-134-96 kubenswrapper[2574]: E0917 17:34:07.395090    2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-96.ec2.internal\" not found"
Sep 17 17:34:07.495581 ip-10-0-134-96 kubenswrapper[2574]: E0917 17:34:07.495544    2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-96.ec2.internal\" not found"
Sep 17 17:34:07.596070 ip-10-0-134-96 kubenswrapper[2574]: E0917 17:34:07.596006    2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-96.ec2.internal\" not found"
Sep 17 17:34:07.611005 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.610987    2574 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 17:34:07.684929 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.684900    2574 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-96.ec2.internal"
Sep 17 17:34:07.694340 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.694321    2574 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 17:34:07.697089 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.697070    2574 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 17 17:34:07.698817 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.698803    2574 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-96.ec2.internal"
Sep 17 17:34:07.705790 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.705775    2574 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 17 17:34:07.738908 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.738873    2574 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 17 17:34:07.739302 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.739014    2574 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 17 17:34:07.739302 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.739056    2574 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 17 17:34:07.739302 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.739065    2574 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 17 17:34:07.765210 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.765189    2574 apiserver.go:52] "Watching apiserver"
Sep 17 17:34:07.784319 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.784297    2574 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 17 17:34:07.787955 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.787935    2574 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 17 17:34:07.788302 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.788283    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-sv2qq","kube-system/kube-apiserver-proxy-ip-10-0-134-96.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rp7jd","openshift-image-registry/node-ca-t79st","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-96.ec2.internal","openshift-multus/multus-ck426","openshift-multus/network-metrics-daemon-4tpm5","openshift-network-diagnostics/network-check-target-j9d2w","kube-system/konnectivity-agent-2j4ds","openshift-cluster-node-tuning-operator/tuned-n28f2","openshift-dns/node-resolver-lf4tf","openshift-multus/multus-additional-cni-plugins-g8nlc","openshift-network-operator/iptables-alerter-xkj8c","openshift-ovn-kubernetes/ovnkube-node-45dp8"]
Sep 17 17:34:07.790786 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.790771    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4tpm5"
Sep 17 17:34:07.790872 ip-10-0-134-96 kubenswrapper[2574]: E0917 17:34:07.790836    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4tpm5" podUID="c18eb096-7089-426a-957b-3dc1a2d7e837"
Sep 17 17:34:07.792818 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.792802    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-t79st"
Sep 17 17:34:07.794863 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.794842    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-xkj8c"
Sep 17 17:34:07.795242 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.795225    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 17 17:34:07.795242 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.795241    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 17 17:34:07.795353 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.795226    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 17 17:34:07.795353 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.795299    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-7s95z\""
Sep 17 17:34:07.796929 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.796914    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rp7jd"
Sep 17 17:34:07.797352 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.797333    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 17 17:34:07.797620 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.797603    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 17 17:34:07.797620 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.797617    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 17 17:34:07.797905 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.797890    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-6dzlz\""
Sep 17 17:34:07.798883 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.798867    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 17 17:34:07.799004 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.798987    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-ck426"
Sep 17 17:34:07.799452 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.799437    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 17 17:34:07.799603 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.799587    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 17 17:34:07.799671 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.799619    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-qgp6d\""
Sep 17 17:34:07.801243 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.801226    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-2j4ds"
Sep 17 17:34:07.801623 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.801607    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 17 17:34:07.801858 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.801846    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 17 17:34:07.802011 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.801998    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 17 17:34:07.802074 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.802016    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 17 17:34:07.802074 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.802019    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-mjm9f\""
Sep 17 17:34:07.803370 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.803355    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sv2qq"
Sep 17 17:34:07.803447 ip-10-0-134-96 kubenswrapper[2574]: E0917 17:34:07.803403    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sv2qq" podUID="fadd68fa-727c-4581-bed0-5ab8b73b9982"
Sep 17 17:34:07.804019 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.804005    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 17 17:34:07.804306 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.804292    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 17 17:34:07.804470 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.804422    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-lslh7\""
Sep 17 17:34:07.805430 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.805414    2574 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 17 17:34:07.805598 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.805582    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-n28f2"
Sep 17 17:34:07.807708 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.807692    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-lf4tf"
Sep 17 17:34:07.808079 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.808067    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 17 17:34:07.808568 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.808552    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-gpmfh\""
Sep 17 17:34:07.808641 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.808571    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 17 17:34:07.809879 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.809862    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-g8nlc"
Sep 17 17:34:07.810211 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.810197    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 17 17:34:07.810211 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.810206    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 17 17:34:07.810303 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.810200    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-jvshz\""
Sep 17 17:34:07.812055 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.812027    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-gvdb9\""
Sep 17 17:34:07.812132 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.812117    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j9d2w"
Sep 17 17:34:07.812196 ip-10-0-134-96 kubenswrapper[2574]: E0917 17:34:07.812174    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j9d2w" podUID="61c43e2e-f36c-401d-a10c-ff5f88fab086"
Sep 17 17:34:07.812305 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.812289    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 17 17:34:07.812455 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.812436    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 17 17:34:07.814568 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.814551    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-45dp8"
Sep 17 17:34:07.816640 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.816625    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-p9w5b\""
Sep 17 17:34:07.816894 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.816882    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 17 17:34:07.817103 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.817090    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 17 17:34:07.817153 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.817137    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 17 17:34:07.817219 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.817203    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 17 17:34:07.819059 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.818559    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 17 17:34:07.819059 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.818574    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 17 17:34:07.824951 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.824913    2574 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-16 17:29:06 +0000 UTC" deadline="2028-02-17 11:32:25.359978926 +0000 UTC"
Sep 17 17:34:07.824951 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.824950    2574 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12425h58m17.535031863s"
Sep 17 17:34:07.831927 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.831912    2574 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-6x5fh"
Sep 17 17:34:07.844067 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.844033    2574 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-6x5fh"
Sep 17 17:34:07.886403 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.886384    2574 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 17 17:34:07.889991 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.889966    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/31495fae-ecfa-4263-a3db-702f75935118-hosts-file\") pod \"node-resolver-lf4tf\" (UID: \"31495fae-ecfa-4263-a3db-702f75935118\") " pod="openshift-dns/node-resolver-lf4tf"
Sep 17 17:34:07.890131 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.890006    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/bd527736-9920-4c79-9069-1eb8be07f869-run-systemd\") pod \"ovnkube-node-45dp8\" (UID: \"bd527736-9920-4c79-9069-1eb8be07f869\") " pod="openshift-ovn-kubernetes/ovnkube-node-45dp8"
Sep 17 17:34:07.890131 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.890033    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ff8b66be-df38-4e3f-9080-483b295fa79c-socket-dir\") pod \"aws-ebs-csi-driver-node-rp7jd\" (UID: \"ff8b66be-df38-4e3f-9080-483b295fa79c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rp7jd"
Sep 17 17:34:07.890131 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.890074    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/94a7bbd1-e769-4046-a56e-442208fb2ca2-run\") pod \"tuned-n28f2\" (UID: \"94a7bbd1-e769-4046-a56e-442208fb2ca2\") " pod="openshift-cluster-node-tuning-operator/tuned-n28f2"
Sep 17 17:34:07.890131 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.890099    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/94a7bbd1-e769-4046-a56e-442208fb2ca2-lib-modules\") pod \"tuned-n28f2\" (UID: \"94a7bbd1-e769-4046-a56e-442208fb2ca2\") " pod="openshift-cluster-node-tuning-operator/tuned-n28f2"
Sep 17 17:34:07.890131 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.890125    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d7f1aae7-8961-4863-abff-c2259ef65ede-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-g8nlc\" (UID: \"d7f1aae7-8961-4863-abff-c2259ef65ede\") " pod="openshift-multus/multus-additional-cni-plugins-g8nlc"
Sep 17 17:34:07.890367 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.890151    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d7f1aae7-8961-4863-abff-c2259ef65ede-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-g8nlc\" (UID: \"d7f1aae7-8961-4863-abff-c2259ef65ede\") " pod="openshift-multus/multus-additional-cni-plugins-g8nlc"
Sep 17 17:34:07.890367 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.890177    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f33e242b-1311-43e7-8a5f-a97ad057236f-host-var-lib-cni-multus\") pod \"multus-ck426\" (UID: \"f33e242b-1311-43e7-8a5f-a97ad057236f\") " pod="openshift-multus/multus-ck426"
Sep 17 17:34:07.890367 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.890200    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fadd68fa-727c-4581-bed0-5ab8b73b9982-original-pull-secret\") pod \"global-pull-secret-syncer-sv2qq\" (UID: \"fadd68fa-727c-4581-bed0-5ab8b73b9982\") " pod="kube-system/global-pull-secret-syncer-sv2qq"
Sep 17 17:34:07.890367 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.890217    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c18eb096-7089-426a-957b-3dc1a2d7e837-metrics-certs\") pod \"network-metrics-daemon-4tpm5\" (UID: \"c18eb096-7089-426a-957b-3dc1a2d7e837\") " pod="openshift-multus/network-metrics-daemon-4tpm5"
Sep 17 17:34:07.890367 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.890272    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/67b5a7ce-5a80-4224-9f54-86ac40fa136b-host\") pod \"node-ca-t79st\" (UID: \"67b5a7ce-5a80-4224-9f54-86ac40fa136b\") " pod="openshift-image-registry/node-ca-t79st"
Sep 17 17:34:07.890367 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.890304    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bd527736-9920-4c79-9069-1eb8be07f869-host-cni-bin\") pod \"ovnkube-node-45dp8\" (UID: \"bd527736-9920-4c79-9069-1eb8be07f869\") " pod="openshift-ovn-kubernetes/ovnkube-node-45dp8"
Sep 17 17:34:07.890367 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.890338    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bd527736-9920-4c79-9069-1eb8be07f869-etc-openvswitch\") pod \"ovnkube-node-45dp8\" (UID: \"bd527736-9920-4c79-9069-1eb8be07f869\") " pod="openshift-ovn-kubernetes/ovnkube-node-45dp8"
Sep 17 17:34:07.890677 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.890374    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bd527736-9920-4c79-9069-1eb8be07f869-run-openvswitch\") pod \"ovnkube-node-45dp8\" (UID: \"bd527736-9920-4c79-9069-1eb8be07f869\") " pod="openshift-ovn-kubernetes/ovnkube-node-45dp8"
Sep 17 17:34:07.890677 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.890416    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/fadd68fa-727c-4581-bed0-5ab8b73b9982-dbus\") pod \"global-pull-secret-syncer-sv2qq\" (UID: \"fadd68fa-727c-4581-bed0-5ab8b73b9982\") " pod="kube-system/global-pull-secret-syncer-sv2qq"
Sep 17 17:34:07.890677 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.890441    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/94a7bbd1-e769-4046-a56e-442208fb2ca2-etc-sysctl-conf\") pod \"tuned-n28f2\" (UID: \"94a7bbd1-e769-4046-a56e-442208fb2ca2\") " pod="openshift-cluster-node-tuning-operator/tuned-n28f2"
Sep 17 17:34:07.890677 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.890522    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d7f1aae7-8961-4863-abff-c2259ef65ede-cnibin\") pod \"multus-additional-cni-plugins-g8nlc\" (UID: \"d7f1aae7-8961-4863-abff-c2259ef65ede\") " pod="openshift-multus/multus-additional-cni-plugins-g8nlc"
Sep 17 17:34:07.890677 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.890560    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bd527736-9920-4c79-9069-1eb8be07f869-host-slash\") pod \"ovnkube-node-45dp8\" (UID: \"bd527736-9920-4c79-9069-1eb8be07f869\") " pod="openshift-ovn-kubernetes/ovnkube-node-45dp8"
Sep 17 17:34:07.890677 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.890586    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/bd527736-9920-4c79-9069-1eb8be07f869-host-cni-netd\") pod \"ovnkube-node-45dp8\" (UID: \"bd527736-9920-4c79-9069-1eb8be07f869\") " pod="openshift-ovn-kubernetes/ovnkube-node-45dp8"
Sep 17 17:34:07.890677 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.890614    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/bd527736-9920-4c79-9069-1eb8be07f869-env-overrides\") pod \"ovnkube-node-45dp8\" (UID: \"bd527736-9920-4c79-9069-1eb8be07f869\") " pod="openshift-ovn-kubernetes/ovnkube-node-45dp8"
Sep 17 17:34:07.890677 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.890639    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f33e242b-1311-43e7-8a5f-a97ad057236f-hostroot\") pod \"multus-ck426\" (UID: \"f33e242b-1311-43e7-8a5f-a97ad057236f\") " pod="openshift-multus/multus-ck426"
Sep 17 17:34:07.890677 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.890655    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f33e242b-1311-43e7-8a5f-a97ad057236f-multus-conf-dir\") pod \"multus-ck426\" (UID: \"f33e242b-1311-43e7-8a5f-a97ad057236f\") " pod="openshift-multus/multus-ck426"
Sep 17 17:34:07.890677 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.890671    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d7f1aae7-8961-4863-abff-c2259ef65ede-tuning-conf-dir\") pod \"multus-additional-cni-plugins-g8nlc\" (UID: \"d7f1aae7-8961-4863-abff-c2259ef65ede\") " pod="openshift-multus/multus-additional-cni-plugins-g8nlc"
Sep 17 17:34:07.891160 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.890695    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/bd527736-9920-4c79-9069-1eb8be07f869-systemd-units\") pod \"ovnkube-node-45dp8\" (UID: \"bd527736-9920-4c79-9069-1eb8be07f869\") " pod="openshift-ovn-kubernetes/ovnkube-node-45dp8"
Sep 17 17:34:07.891160 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.890709    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bd527736-9920-4c79-9069-1eb8be07f869-var-lib-openvswitch\") pod \"ovnkube-node-45dp8\" (UID: \"bd527736-9920-4c79-9069-1eb8be07f869\") " pod="openshift-ovn-kubernetes/ovnkube-node-45dp8"
Sep 17 17:34:07.891160 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.890723    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ff8b66be-df38-4e3f-9080-483b295fa79c-device-dir\") pod \"aws-ebs-csi-driver-node-rp7jd\" (UID: \"ff8b66be-df38-4e3f-9080-483b295fa79c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rp7jd"
Sep 17 17:34:07.891160 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.890738    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tgm22\" (UniqueName: \"kubernetes.io/projected/ff8b66be-df38-4e3f-9080-483b295fa79c-kube-api-access-tgm22\") pod \"aws-ebs-csi-driver-node-rp7jd\" (UID: \"ff8b66be-df38-4e3f-9080-483b295fa79c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rp7jd"
Sep 17 17:34:07.891160 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.890755    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/94a7bbd1-e769-4046-a56e-442208fb2ca2-etc-tuned\") pod \"tuned-n28f2\" (UID: \"94a7bbd1-e769-4046-a56e-442208fb2ca2\") " pod="openshift-cluster-node-tuning-operator/tuned-n28f2"
Sep 17 17:34:07.891160 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.890808    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d7f1aae7-8961-4863-abff-c2259ef65ede-cni-binary-copy\") pod \"multus-additional-cni-plugins-g8nlc\" (UID: \"d7f1aae7-8961-4863-abff-c2259ef65ede\") " pod="openshift-multus/multus-additional-cni-plugins-g8nlc"
Sep 17 17:34:07.891160 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.890837    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/fadd68fa-727c-4581-bed0-5ab8b73b9982-kubelet-config\") pod \"global-pull-secret-syncer-sv2qq\" (UID: \"fadd68fa-727c-4581-bed0-5ab8b73b9982\") " pod="kube-system/global-pull-secret-syncer-sv2qq"
Sep 17 17:34:07.891160 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.890856    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/94a7bbd1-e769-4046-a56e-442208fb2ca2-etc-kubernetes\") pod \"tuned-n28f2\" (UID: \"94a7bbd1-e769-4046-a56e-442208fb2ca2\") " pod="openshift-cluster-node-tuning-operator/tuned-n28f2"
Sep 17 17:34:07.891160 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.890869    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/94a7bbd1-e769-4046-a56e-442208fb2ca2-etc-sysctl-d\") pod \"tuned-n28f2\" (UID: \"94a7bbd1-e769-4046-a56e-442208fb2ca2\") " pod="openshift-cluster-node-tuning-operator/tuned-n28f2"
Sep 17 17:34:07.891160 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.890883    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/94a7bbd1-e769-4046-a56e-442208fb2ca2-tmp\") pod \"tuned-n28f2\" (UID: \"94a7bbd1-e769-4046-a56e-442208fb2ca2\") " pod="openshift-cluster-node-tuning-operator/tuned-n28f2"
Sep 17 17:34:07.891160 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.890902    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bd527736-9920-4c79-9069-1eb8be07f869-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-45dp8\" (UID: \"bd527736-9920-4c79-9069-1eb8be07f869\") " pod="openshift-ovn-kubernetes/ovnkube-node-45dp8"
Sep 17 17:34:07.891160 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.890927    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bd527736-9920-4c79-9069-1eb8be07f869-host-run-netns\") pod \"ovnkube-node-45dp8\" (UID: \"bd527736-9920-4c79-9069-1eb8be07f869\") " pod="openshift-ovn-kubernetes/ovnkube-node-45dp8"
Sep 17 17:34:07.891160 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.890945    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ff8b66be-df38-4e3f-9080-483b295fa79c-registration-dir\") pod \"aws-ebs-csi-driver-node-rp7jd\" (UID: \"ff8b66be-df38-4e3f-9080-483b295fa79c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rp7jd"
Sep 17 17:34:07.891160 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.890962    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f33e242b-1311-43e7-8a5f-a97ad057236f-system-cni-dir\") pod \"multus-ck426\" (UID: \"f33e242b-1311-43e7-8a5f-a97ad057236f\") " pod="openshift-multus/multus-ck426"
Sep 17 17:34:07.891160 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.890993    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f33e242b-1311-43e7-8a5f-a97ad057236f-host-run-netns\") pod \"multus-ck426\" (UID: \"f33e242b-1311-43e7-8a5f-a97ad057236f\") " pod="openshift-multus/multus-ck426"
Sep 17 17:34:07.891160 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.891012    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d3f553e8-b416-4d7d-bbef-991a0404b287-agent-certs\") pod \"konnectivity-agent-2j4ds\" (UID: \"d3f553e8-b416-4d7d-bbef-991a0404b287\") " pod="kube-system/konnectivity-agent-2j4ds"
Sep 17 17:34:07.891869 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.891025    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/94a7bbd1-e769-4046-a56e-442208fb2ca2-host\") pod \"tuned-n28f2\" (UID: \"94a7bbd1-e769-4046-a56e-442208fb2ca2\") " pod="openshift-cluster-node-tuning-operator/tuned-n28f2"
Sep 17 17:34:07.891869 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.891062    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bsq5x\" (UniqueName: \"kubernetes.io/projected/31495fae-ecfa-4263-a3db-702f75935118-kube-api-access-bsq5x\") pod \"node-resolver-lf4tf\" (UID: \"31495fae-ecfa-4263-a3db-702f75935118\") " pod="openshift-dns/node-resolver-lf4tf"
Sep 17 17:34:07.891869 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.891088    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-894mr\" (UniqueName: \"kubernetes.io/projected/d7f1aae7-8961-4863-abff-c2259ef65ede-kube-api-access-894mr\") pod \"multus-additional-cni-plugins-g8nlc\" (UID: \"d7f1aae7-8961-4863-abff-c2259ef65ede\") " pod="openshift-multus/multus-additional-cni-plugins-g8nlc"
Sep 17 17:34:07.891869 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.891103    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/94a7bbd1-e769-4046-a56e-442208fb2ca2-var-lib-kubelet\") pod \"tuned-n28f2\" (UID: \"94a7bbd1-e769-4046-a56e-442208fb2ca2\") " pod="openshift-cluster-node-tuning-operator/tuned-n28f2"
Sep 17 17:34:07.891869 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.891116    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-smzws\" (UniqueName: \"kubernetes.io/projected/bd527736-9920-4c79-9069-1eb8be07f869-kube-api-access-smzws\") pod \"ovnkube-node-45dp8\" (UID: \"bd527736-9920-4c79-9069-1eb8be07f869\") " pod="openshift-ovn-kubernetes/ovnkube-node-45dp8"
Sep 17 17:34:07.891869 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.891142    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f33e242b-1311-43e7-8a5f-a97ad057236f-multus-cni-dir\") pod \"multus-ck426\" (UID: \"f33e242b-1311-43e7-8a5f-a97ad057236f\") " pod="openshift-multus/multus-ck426"
Sep 17 17:34:07.891869 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.891161    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f33e242b-1311-43e7-8a5f-a97ad057236f-host-var-lib-kubelet\") pod \"multus-ck426\" (UID: \"f33e242b-1311-43e7-8a5f-a97ad057236f\") " pod="openshift-multus/multus-ck426"
Sep 17 17:34:07.891869 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.891176    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/12264c60-6721-4820-816f-798eefa04bd5-iptables-alerter-script\") pod \"iptables-alerter-xkj8c\" (UID: \"12264c60-6721-4820-816f-798eefa04bd5\") " pod="openshift-network-operator/iptables-alerter-xkj8c"
Sep 17 17:34:07.891869 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.891192    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mbbqs\" (UniqueName: \"kubernetes.io/projected/12264c60-6721-4820-816f-798eefa04bd5-kube-api-access-mbbqs\") pod \"iptables-alerter-xkj8c\" (UID: \"12264c60-6721-4820-816f-798eefa04bd5\") " pod="openshift-network-operator/iptables-alerter-xkj8c"
Sep 17 17:34:07.891869 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.891216    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/94a7bbd1-e769-4046-a56e-442208fb2ca2-etc-modprobe-d\") pod \"tuned-n28f2\" (UID: \"94a7bbd1-e769-4046-a56e-442208fb2ca2\") " pod="openshift-cluster-node-tuning-operator/tuned-n28f2"
Sep 17 17:34:07.891869 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.891237    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/94a7bbd1-e769-4046-a56e-442208fb2ca2-etc-sysconfig\") pod \"tuned-n28f2\" (UID: \"94a7bbd1-e769-4046-a56e-442208fb2ca2\") " pod="openshift-cluster-node-tuning-operator/tuned-n28f2"
Sep 17 17:34:07.891869 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.891257    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/bd527736-9920-4c79-9069-1eb8be07f869-log-socket\") pod \"ovnkube-node-45dp8\" (UID: \"bd527736-9920-4c79-9069-1eb8be07f869\") " pod="openshift-ovn-kubernetes/ovnkube-node-45dp8"
Sep 17 17:34:07.891869 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.891306    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/bd527736-9920-4c79-9069-1eb8be07f869-ovnkube-script-lib\") pod \"ovnkube-node-45dp8\" (UID: \"bd527736-9920-4c79-9069-1eb8be07f869\") " pod="openshift-ovn-kubernetes/ovnkube-node-45dp8"
Sep 17 17:34:07.891869 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.891337    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f33e242b-1311-43e7-8a5f-a97ad057236f-multus-socket-dir-parent\") pod \"multus-ck426\" (UID: \"f33e242b-1311-43e7-8a5f-a97ad057236f\") " pod="openshift-multus/multus-ck426"
Sep 17 17:34:07.891869 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.891361    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f33e242b-1311-43e7-8a5f-a97ad057236f-host-run-multus-certs\") pod \"multus-ck426\" (UID: \"f33e242b-1311-43e7-8a5f-a97ad057236f\") " pod="openshift-multus/multus-ck426"
Sep 17 17:34:07.891869 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.891400    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d3f553e8-b416-4d7d-bbef-991a0404b287-konnectivity-ca\") pod \"konnectivity-agent-2j4ds\" (UID: \"d3f553e8-b416-4d7d-bbef-991a0404b287\") " pod="kube-system/konnectivity-agent-2j4ds"
Sep 17 17:34:07.892622 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.891425    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/67b5a7ce-5a80-4224-9f54-86ac40fa136b-serviceca\") pod \"node-ca-t79st\" (UID: \"67b5a7ce-5a80-4224-9f54-86ac40fa136b\") " pod="openshift-image-registry/node-ca-t79st"
Sep 17 17:34:07.892622 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.891449    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tp8c8\" (UniqueName: \"kubernetes.io/projected/67b5a7ce-5a80-4224-9f54-86ac40fa136b-kube-api-access-tp8c8\") pod \"node-ca-t79st\" (UID: \"67b5a7ce-5a80-4224-9f54-86ac40fa136b\") " pod="openshift-image-registry/node-ca-t79st"
Sep 17 17:34:07.892622 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.891471    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vz647\" (UniqueName: \"kubernetes.io/projected/94a7bbd1-e769-4046-a56e-442208fb2ca2-kube-api-access-vz647\") pod \"tuned-n28f2\" (UID: \"94a7bbd1-e769-4046-a56e-442208fb2ca2\") " pod="openshift-cluster-node-tuning-operator/tuned-n28f2"
Sep 17 17:34:07.892622 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.891485    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/bd527736-9920-4c79-9069-1eb8be07f869-ovnkube-config\") pod \"ovnkube-node-45dp8\" (UID: \"bd527736-9920-4c79-9069-1eb8be07f869\") " pod="openshift-ovn-kubernetes/ovnkube-node-45dp8"
Sep 17 17:34:07.892622 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.891498    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/bd527736-9920-4c79-9069-1eb8be07f869-ovn-node-metrics-cert\") pod \"ovnkube-node-45dp8\" (UID: \"bd527736-9920-4c79-9069-1eb8be07f869\") " pod="openshift-ovn-kubernetes/ovnkube-node-45dp8"
Sep 17 17:34:07.892622 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.891511    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f33e242b-1311-43e7-8a5f-a97ad057236f-host-run-k8s-cni-cncf-io\") pod \"multus-ck426\" (UID: \"f33e242b-1311-43e7-8a5f-a97ad057236f\") " pod="openshift-multus/multus-ck426"
Sep 17 17:34:07.892622 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.891525    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ff8b66be-df38-4e3f-9080-483b295fa79c-sys-fs\") pod \"aws-ebs-csi-driver-node-rp7jd\" (UID: \"ff8b66be-df38-4e3f-9080-483b295fa79c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rp7jd"
Sep 17 17:34:07.892622 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.891567    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f33e242b-1311-43e7-8a5f-a97ad057236f-cnibin\") pod \"multus-ck426\" (UID: \"f33e242b-1311-43e7-8a5f-a97ad057236f\") " pod="openshift-multus/multus-ck426"
Sep 17 17:34:07.892622 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.891601    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f33e242b-1311-43e7-8a5f-a97ad057236f-host-var-lib-cni-bin\") pod \"multus-ck426\" (UID: \"f33e242b-1311-43e7-8a5f-a97ad057236f\") " pod="openshift-multus/multus-ck426"
Sep 17 17:34:07.892622 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.891636    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/12264c60-6721-4820-816f-798eefa04bd5-host-slash\") pod \"iptables-alerter-xkj8c\" (UID: \"12264c60-6721-4820-816f-798eefa04bd5\") " pod="openshift-network-operator/iptables-alerter-xkj8c"
Sep 17 17:34:07.892622 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.891662    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/f33e242b-1311-43e7-8a5f-a97ad057236f-multus-daemon-config\") pod \"multus-ck426\" (UID: \"f33e242b-1311-43e7-8a5f-a97ad057236f\") " pod="openshift-multus/multus-ck426"
Sep 17 17:34:07.892622 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.891677    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/bd527736-9920-4c79-9069-1eb8be07f869-run-ovn\") pod \"ovnkube-node-45dp8\" (UID: \"bd527736-9920-4c79-9069-1eb8be07f869\") " pod="openshift-ovn-kubernetes/ovnkube-node-45dp8"
Sep 17 17:34:07.892622 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.891709    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ff8b66be-df38-4e3f-9080-483b295fa79c-etc-selinux\") pod \"aws-ebs-csi-driver-node-rp7jd\" (UID: \"ff8b66be-df38-4e3f-9080-483b295fa79c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rp7jd"
Sep 17 17:34:07.892622 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.891730    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f33e242b-1311-43e7-8a5f-a97ad057236f-etc-kubernetes\") pod \"multus-ck426\" (UID: \"f33e242b-1311-43e7-8a5f-a97ad057236f\") " pod="openshift-multus/multus-ck426"
Sep 17 17:34:07.892622 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.891746    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/31495fae-ecfa-4263-a3db-702f75935118-tmp-dir\") pod \"node-resolver-lf4tf\" (UID: \"31495fae-ecfa-4263-a3db-702f75935118\") " pod="openshift-dns/node-resolver-lf4tf"
Sep 17 17:34:07.892622 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.891760    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/bd527736-9920-4c79-9069-1eb8be07f869-host-kubelet\") pod \"ovnkube-node-45dp8\" (UID: \"bd527736-9920-4c79-9069-1eb8be07f869\") " pod="openshift-ovn-kubernetes/ovnkube-node-45dp8"
Sep 17 17:34:07.892622 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.891779    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/bd527736-9920-4c79-9069-1eb8be07f869-node-log\") pod \"ovnkube-node-45dp8\" (UID: \"bd527736-9920-4c79-9069-1eb8be07f869\") " pod="openshift-ovn-kubernetes/ovnkube-node-45dp8"
Sep 17 17:34:07.893509 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.891800    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ff8b66be-df38-4e3f-9080-483b295fa79c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-rp7jd\" (UID: \"ff8b66be-df38-4e3f-9080-483b295fa79c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rp7jd"
Sep 17 17:34:07.893509 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.891816    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2lk56\" (UniqueName: \"kubernetes.io/projected/61c43e2e-f36c-401d-a10c-ff5f88fab086-kube-api-access-2lk56\") pod \"network-check-target-j9d2w\" (UID: \"61c43e2e-f36c-401d-a10c-ff5f88fab086\") " pod="openshift-network-diagnostics/network-check-target-j9d2w"
Sep 17 17:34:07.893509 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.891837    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mk2pz\" (UniqueName: \"kubernetes.io/projected/c18eb096-7089-426a-957b-3dc1a2d7e837-kube-api-access-mk2pz\") pod \"network-metrics-daemon-4tpm5\" (UID: \"c18eb096-7089-426a-957b-3dc1a2d7e837\") " pod="openshift-multus/network-metrics-daemon-4tpm5"
Sep 17 17:34:07.893509 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.891856    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/94a7bbd1-e769-4046-a56e-442208fb2ca2-sys\") pod \"tuned-n28f2\" (UID: \"94a7bbd1-e769-4046-a56e-442208fb2ca2\") " pod="openshift-cluster-node-tuning-operator/tuned-n28f2"
Sep 17 17:34:07.893509 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.891875    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d7f1aae7-8961-4863-abff-c2259ef65ede-system-cni-dir\") pod \"multus-additional-cni-plugins-g8nlc\" (UID: \"d7f1aae7-8961-4863-abff-c2259ef65ede\") " pod="openshift-multus/multus-additional-cni-plugins-g8nlc"
Sep 17 17:34:07.893509 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.891889    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bd527736-9920-4c79-9069-1eb8be07f869-host-run-ovn-kubernetes\") pod \"ovnkube-node-45dp8\" (UID: \"bd527736-9920-4c79-9069-1eb8be07f869\") " pod="openshift-ovn-kubernetes/ovnkube-node-45dp8"
Sep 17 17:34:07.893509 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.891922    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f33e242b-1311-43e7-8a5f-a97ad057236f-cni-binary-copy\") pod \"multus-ck426\" (UID: \"f33e242b-1311-43e7-8a5f-a97ad057236f\") " pod="openshift-multus/multus-ck426"
Sep 17 17:34:07.893509 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.891941    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h26rj\" (UniqueName: \"kubernetes.io/projected/f33e242b-1311-43e7-8a5f-a97ad057236f-kube-api-access-h26rj\") pod \"multus-ck426\" (UID: \"f33e242b-1311-43e7-8a5f-a97ad057236f\") " pod="openshift-multus/multus-ck426"
Sep 17 17:34:07.893509 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.891975    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/94a7bbd1-e769-4046-a56e-442208fb2ca2-etc-systemd\") pod \"tuned-n28f2\" (UID: \"94a7bbd1-e769-4046-a56e-442208fb2ca2\") " pod="openshift-cluster-node-tuning-operator/tuned-n28f2"
Sep 17 17:34:07.993130 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.993105    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ff8b66be-df38-4e3f-9080-483b295fa79c-sys-fs\") pod \"aws-ebs-csi-driver-node-rp7jd\" (UID: \"ff8b66be-df38-4e3f-9080-483b295fa79c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rp7jd"
Sep 17 17:34:07.993246 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.993136    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f33e242b-1311-43e7-8a5f-a97ad057236f-cnibin\") pod \"multus-ck426\" (UID: \"f33e242b-1311-43e7-8a5f-a97ad057236f\") " pod="openshift-multus/multus-ck426"
Sep 17 17:34:07.993246 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.993151    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f33e242b-1311-43e7-8a5f-a97ad057236f-host-var-lib-cni-bin\") pod \"multus-ck426\" (UID: \"f33e242b-1311-43e7-8a5f-a97ad057236f\") " pod="openshift-multus/multus-ck426"
Sep 17 17:34:07.993246 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.993165    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/12264c60-6721-4820-816f-798eefa04bd5-host-slash\") pod \"iptables-alerter-xkj8c\" (UID: \"12264c60-6721-4820-816f-798eefa04bd5\") " pod="openshift-network-operator/iptables-alerter-xkj8c"
Sep 17 17:34:07.993246 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.993206    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/12264c60-6721-4820-816f-798eefa04bd5-host-slash\") pod \"iptables-alerter-xkj8c\" (UID: \"12264c60-6721-4820-816f-798eefa04bd5\") " pod="openshift-network-operator/iptables-alerter-xkj8c"
Sep 17 17:34:07.993246 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.993205    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/f33e242b-1311-43e7-8a5f-a97ad057236f-multus-daemon-config\") pod \"multus-ck426\" (UID: \"f33e242b-1311-43e7-8a5f-a97ad057236f\") " pod="openshift-multus/multus-ck426"
Sep 17 17:34:07.993246 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.993229    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f33e242b-1311-43e7-8a5f-a97ad057236f-host-var-lib-cni-bin\") pod \"multus-ck426\" (UID: \"f33e242b-1311-43e7-8a5f-a97ad057236f\") " pod="openshift-multus/multus-ck426"
Sep 17 17:34:07.993246 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.993241    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/bd527736-9920-4c79-9069-1eb8be07f869-run-ovn\") pod \"ovnkube-node-45dp8\" (UID: \"bd527736-9920-4c79-9069-1eb8be07f869\") " pod="openshift-ovn-kubernetes/ovnkube-node-45dp8"
Sep 17 17:34:07.993246 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.993245    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f33e242b-1311-43e7-8a5f-a97ad057236f-cnibin\") pod \"multus-ck426\" (UID: \"f33e242b-1311-43e7-8a5f-a97ad057236f\") " pod="openshift-multus/multus-ck426"
Sep 17 17:34:07.993636 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.993244    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ff8b66be-df38-4e3f-9080-483b295fa79c-sys-fs\") pod \"aws-ebs-csi-driver-node-rp7jd\" (UID: \"ff8b66be-df38-4e3f-9080-483b295fa79c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rp7jd"
Sep 17 17:34:07.993636 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.993282    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ff8b66be-df38-4e3f-9080-483b295fa79c-etc-selinux\") pod \"aws-ebs-csi-driver-node-rp7jd\" (UID: \"ff8b66be-df38-4e3f-9080-483b295fa79c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rp7jd"
Sep 17 17:34:07.993636 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.993290    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/bd527736-9920-4c79-9069-1eb8be07f869-run-ovn\") pod \"ovnkube-node-45dp8\" (UID: \"bd527736-9920-4c79-9069-1eb8be07f869\") " pod="openshift-ovn-kubernetes/ovnkube-node-45dp8"
Sep 17 17:34:07.993636 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.993309    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f33e242b-1311-43e7-8a5f-a97ad057236f-etc-kubernetes\") pod \"multus-ck426\" (UID: \"f33e242b-1311-43e7-8a5f-a97ad057236f\") " pod="openshift-multus/multus-ck426"
Sep 17 17:34:07.993636 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.993333    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/31495fae-ecfa-4263-a3db-702f75935118-tmp-dir\") pod \"node-resolver-lf4tf\" (UID: \"31495fae-ecfa-4263-a3db-702f75935118\") " pod="openshift-dns/node-resolver-lf4tf"
Sep 17 17:34:07.993636 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.993356    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/bd527736-9920-4c79-9069-1eb8be07f869-host-kubelet\") pod \"ovnkube-node-45dp8\" (UID: \"bd527736-9920-4c79-9069-1eb8be07f869\") " pod="openshift-ovn-kubernetes/ovnkube-node-45dp8"
Sep 17 17:34:07.993636 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.993368    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ff8b66be-df38-4e3f-9080-483b295fa79c-etc-selinux\") pod \"aws-ebs-csi-driver-node-rp7jd\" (UID: \"ff8b66be-df38-4e3f-9080-483b295fa79c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rp7jd"
Sep 17 17:34:07.993636 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.993378    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/bd527736-9920-4c79-9069-1eb8be07f869-node-log\") pod \"ovnkube-node-45dp8\" (UID: \"bd527736-9920-4c79-9069-1eb8be07f869\") " pod="openshift-ovn-kubernetes/ovnkube-node-45dp8"
Sep 17 17:34:07.993636 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.993400    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ff8b66be-df38-4e3f-9080-483b295fa79c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-rp7jd\" (UID: \"ff8b66be-df38-4e3f-9080-483b295fa79c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rp7jd"
Sep 17 17:34:07.993636 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.993432    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/bd527736-9920-4c79-9069-1eb8be07f869-host-kubelet\") pod \"ovnkube-node-45dp8\" (UID: \"bd527736-9920-4c79-9069-1eb8be07f869\") " pod="openshift-ovn-kubernetes/ovnkube-node-45dp8"
Sep 17 17:34:07.993636 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.993453    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/bd527736-9920-4c79-9069-1eb8be07f869-node-log\") pod \"ovnkube-node-45dp8\" (UID: \"bd527736-9920-4c79-9069-1eb8be07f869\") " pod="openshift-ovn-kubernetes/ovnkube-node-45dp8"
Sep 17 17:34:07.993636 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.993435    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f33e242b-1311-43e7-8a5f-a97ad057236f-etc-kubernetes\") pod \"multus-ck426\" (UID: \"f33e242b-1311-43e7-8a5f-a97ad057236f\") " pod="openshift-multus/multus-ck426"
Sep 17 17:34:07.993636 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.993477    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2lk56\" (UniqueName: \"kubernetes.io/projected/61c43e2e-f36c-401d-a10c-ff5f88fab086-kube-api-access-2lk56\") pod \"network-check-target-j9d2w\" (UID: \"61c43e2e-f36c-401d-a10c-ff5f88fab086\") " pod="openshift-network-diagnostics/network-check-target-j9d2w"
Sep 17 17:34:07.993636 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.993507    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mk2pz\" (UniqueName: \"kubernetes.io/projected/c18eb096-7089-426a-957b-3dc1a2d7e837-kube-api-access-mk2pz\") pod \"network-metrics-daemon-4tpm5\" (UID: \"c18eb096-7089-426a-957b-3dc1a2d7e837\") " pod="openshift-multus/network-metrics-daemon-4tpm5"
Sep 17 17:34:07.993636 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.993517    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ff8b66be-df38-4e3f-9080-483b295fa79c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-rp7jd\" (UID: \"ff8b66be-df38-4e3f-9080-483b295fa79c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rp7jd"
Sep 17 17:34:07.993636 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.993532    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/94a7bbd1-e769-4046-a56e-442208fb2ca2-sys\") pod \"tuned-n28f2\" (UID: \"94a7bbd1-e769-4046-a56e-442208fb2ca2\") " pod="openshift-cluster-node-tuning-operator/tuned-n28f2"
Sep 17 17:34:07.993636 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.993555    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d7f1aae7-8961-4863-abff-c2259ef65ede-system-cni-dir\") pod \"multus-additional-cni-plugins-g8nlc\" (UID: \"d7f1aae7-8961-4863-abff-c2259ef65ede\") " pod="openshift-multus/multus-additional-cni-plugins-g8nlc"
Sep 17 17:34:07.994512 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.993603    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/94a7bbd1-e769-4046-a56e-442208fb2ca2-sys\") pod \"tuned-n28f2\" (UID: \"94a7bbd1-e769-4046-a56e-442208fb2ca2\") " pod="openshift-cluster-node-tuning-operator/tuned-n28f2"
Sep 17 17:34:07.994512 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.993634    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d7f1aae7-8961-4863-abff-c2259ef65ede-system-cni-dir\") pod \"multus-additional-cni-plugins-g8nlc\" (UID: \"d7f1aae7-8961-4863-abff-c2259ef65ede\") " pod="openshift-multus/multus-additional-cni-plugins-g8nlc"
Sep 17 17:34:07.994512 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.993660    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/31495fae-ecfa-4263-a3db-702f75935118-tmp-dir\") pod \"node-resolver-lf4tf\" (UID: \"31495fae-ecfa-4263-a3db-702f75935118\") " pod="openshift-dns/node-resolver-lf4tf"
Sep 17 17:34:07.994512 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.993689    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/f33e242b-1311-43e7-8a5f-a97ad057236f-multus-daemon-config\") pod \"multus-ck426\" (UID: \"f33e242b-1311-43e7-8a5f-a97ad057236f\") " pod="openshift-multus/multus-ck426"
Sep 17 17:34:07.994512 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.993743    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bd527736-9920-4c79-9069-1eb8be07f869-host-run-ovn-kubernetes\") pod \"ovnkube-node-45dp8\" (UID: \"bd527736-9920-4c79-9069-1eb8be07f869\") " pod="openshift-ovn-kubernetes/ovnkube-node-45dp8"
Sep 17 17:34:07.994512 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.993608    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bd527736-9920-4c79-9069-1eb8be07f869-host-run-ovn-kubernetes\") pod \"ovnkube-node-45dp8\" (UID: \"bd527736-9920-4c79-9069-1eb8be07f869\") " pod="openshift-ovn-kubernetes/ovnkube-node-45dp8"
Sep 17 17:34:07.994512 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.994149    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f33e242b-1311-43e7-8a5f-a97ad057236f-cni-binary-copy\") pod \"multus-ck426\" (UID: \"f33e242b-1311-43e7-8a5f-a97ad057236f\") " pod="openshift-multus/multus-ck426"
Sep 17 17:34:07.994512 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.994178    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h26rj\" (UniqueName: \"kubernetes.io/projected/f33e242b-1311-43e7-8a5f-a97ad057236f-kube-api-access-h26rj\") pod \"multus-ck426\" (UID: \"f33e242b-1311-43e7-8a5f-a97ad057236f\") " pod="openshift-multus/multus-ck426"
Sep 17 17:34:07.994512 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.994201    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/94a7bbd1-e769-4046-a56e-442208fb2ca2-etc-systemd\") pod \"tuned-n28f2\" (UID: \"94a7bbd1-e769-4046-a56e-442208fb2ca2\") " pod="openshift-cluster-node-tuning-operator/tuned-n28f2"
Sep 17 17:34:07.994512 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.994224    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/31495fae-ecfa-4263-a3db-702f75935118-hosts-file\") pod \"node-resolver-lf4tf\" (UID: \"31495fae-ecfa-4263-a3db-702f75935118\") " pod="openshift-dns/node-resolver-lf4tf"
Sep 17 17:34:07.994512 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.994246    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/bd527736-9920-4c79-9069-1eb8be07f869-run-systemd\") pod \"ovnkube-node-45dp8\" (UID: \"bd527736-9920-4c79-9069-1eb8be07f869\") " pod="openshift-ovn-kubernetes/ovnkube-node-45dp8"
Sep 17 17:34:07.994512 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.994270    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ff8b66be-df38-4e3f-9080-483b295fa79c-socket-dir\") pod \"aws-ebs-csi-driver-node-rp7jd\" (UID: \"ff8b66be-df38-4e3f-9080-483b295fa79c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rp7jd"
Sep 17 17:34:07.994512 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.994296    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/94a7bbd1-e769-4046-a56e-442208fb2ca2-run\") pod \"tuned-n28f2\" (UID: \"94a7bbd1-e769-4046-a56e-442208fb2ca2\") " pod="openshift-cluster-node-tuning-operator/tuned-n28f2"
Sep 17 17:34:07.994512 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.994268    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/94a7bbd1-e769-4046-a56e-442208fb2ca2-etc-systemd\") pod \"tuned-n28f2\" (UID: \"94a7bbd1-e769-4046-a56e-442208fb2ca2\") " pod="openshift-cluster-node-tuning-operator/tuned-n28f2"
Sep 17 17:34:07.994512 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.994318    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/94a7bbd1-e769-4046-a56e-442208fb2ca2-lib-modules\") pod \"tuned-n28f2\" (UID: \"94a7bbd1-e769-4046-a56e-442208fb2ca2\") " pod="openshift-cluster-node-tuning-operator/tuned-n28f2"
Sep 17 17:34:07.994512 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.994319    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/31495fae-ecfa-4263-a3db-702f75935118-hosts-file\") pod \"node-resolver-lf4tf\" (UID: \"31495fae-ecfa-4263-a3db-702f75935118\") " pod="openshift-dns/node-resolver-lf4tf"
Sep 17 17:34:07.994512 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.994347    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/bd527736-9920-4c79-9069-1eb8be07f869-run-systemd\") pod \"ovnkube-node-45dp8\" (UID: \"bd527736-9920-4c79-9069-1eb8be07f869\") " pod="openshift-ovn-kubernetes/ovnkube-node-45dp8"
Sep 17 17:34:07.994512 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.994363    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d7f1aae7-8961-4863-abff-c2259ef65ede-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-g8nlc\" (UID: \"d7f1aae7-8961-4863-abff-c2259ef65ede\") " pod="openshift-multus/multus-additional-cni-plugins-g8nlc"
Sep 17 17:34:07.995338 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.994373    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ff8b66be-df38-4e3f-9080-483b295fa79c-socket-dir\") pod \"aws-ebs-csi-driver-node-rp7jd\" (UID: \"ff8b66be-df38-4e3f-9080-483b295fa79c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rp7jd"
Sep 17 17:34:07.995338 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.994427    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/94a7bbd1-e769-4046-a56e-442208fb2ca2-run\") pod \"tuned-n28f2\" (UID: \"94a7bbd1-e769-4046-a56e-442208fb2ca2\") " pod="openshift-cluster-node-tuning-operator/tuned-n28f2"
Sep 17 17:34:07.995338 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.994462    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d7f1aae7-8961-4863-abff-c2259ef65ede-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-g8nlc\" (UID: \"d7f1aae7-8961-4863-abff-c2259ef65ede\") " pod="openshift-multus/multus-additional-cni-plugins-g8nlc"
Sep 17 17:34:07.995338 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.994487    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/94a7bbd1-e769-4046-a56e-442208fb2ca2-lib-modules\") pod \"tuned-n28f2\" (UID: \"94a7bbd1-e769-4046-a56e-442208fb2ca2\") " pod="openshift-cluster-node-tuning-operator/tuned-n28f2"
Sep 17 17:34:07.995338 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.994491    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f33e242b-1311-43e7-8a5f-a97ad057236f-host-var-lib-cni-multus\") pod \"multus-ck426\" (UID: \"f33e242b-1311-43e7-8a5f-a97ad057236f\") " pod="openshift-multus/multus-ck426"
Sep 17 17:34:07.995338 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.994539    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fadd68fa-727c-4581-bed0-5ab8b73b9982-original-pull-secret\") pod \"global-pull-secret-syncer-sv2qq\" (UID: \"fadd68fa-727c-4581-bed0-5ab8b73b9982\") " pod="kube-system/global-pull-secret-syncer-sv2qq"
Sep 17 17:34:07.995338 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.994565    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c18eb096-7089-426a-957b-3dc1a2d7e837-metrics-certs\") pod \"network-metrics-daemon-4tpm5\" (UID: \"c18eb096-7089-426a-957b-3dc1a2d7e837\") " pod="openshift-multus/network-metrics-daemon-4tpm5"
Sep 17 17:34:07.995338 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.994588    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/67b5a7ce-5a80-4224-9f54-86ac40fa136b-host\") pod \"node-ca-t79st\" (UID: \"67b5a7ce-5a80-4224-9f54-86ac40fa136b\") " pod="openshift-image-registry/node-ca-t79st"
Sep 17 17:34:07.995338 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.994610    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bd527736-9920-4c79-9069-1eb8be07f869-host-cni-bin\") pod \"ovnkube-node-45dp8\" (UID: \"bd527736-9920-4c79-9069-1eb8be07f869\") " pod="openshift-ovn-kubernetes/ovnkube-node-45dp8"
Sep 17 17:34:07.995338 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.994619    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f33e242b-1311-43e7-8a5f-a97ad057236f-host-var-lib-cni-multus\") pod \"multus-ck426\" (UID: \"f33e242b-1311-43e7-8a5f-a97ad057236f\") " pod="openshift-multus/multus-ck426"
Sep 17 17:34:07.995338 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.994620    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f33e242b-1311-43e7-8a5f-a97ad057236f-cni-binary-copy\") pod \"multus-ck426\" (UID: \"f33e242b-1311-43e7-8a5f-a97ad057236f\") " pod="openshift-multus/multus-ck426"
Sep 17 17:34:07.995338 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.994635    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bd527736-9920-4c79-9069-1eb8be07f869-etc-openvswitch\") pod \"ovnkube-node-45dp8\" (UID: \"bd527736-9920-4c79-9069-1eb8be07f869\") " pod="openshift-ovn-kubernetes/ovnkube-node-45dp8"
Sep 17 17:34:07.995338 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.994666    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bd527736-9920-4c79-9069-1eb8be07f869-etc-openvswitch\") pod \"ovnkube-node-45dp8\" (UID: \"bd527736-9920-4c79-9069-1eb8be07f869\") " pod="openshift-ovn-kubernetes/ovnkube-node-45dp8"
Sep 17 17:34:07.995338 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.994670    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bd527736-9920-4c79-9069-1eb8be07f869-run-openvswitch\") pod \"ovnkube-node-45dp8\" (UID: \"bd527736-9920-4c79-9069-1eb8be07f869\") " pod="openshift-ovn-kubernetes/ovnkube-node-45dp8"
Sep 17 17:34:07.995338 ip-10-0-134-96 kubenswrapper[2574]: E0917 17:34:07.994685    2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:34:07.995338 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.994698    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/fadd68fa-727c-4581-bed0-5ab8b73b9982-dbus\") pod \"global-pull-secret-syncer-sv2qq\" (UID: \"fadd68fa-727c-4581-bed0-5ab8b73b9982\") " pod="kube-system/global-pull-secret-syncer-sv2qq"
Sep 17 17:34:07.995338 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.994705    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bd527736-9920-4c79-9069-1eb8be07f869-run-openvswitch\") pod \"ovnkube-node-45dp8\" (UID: \"bd527736-9920-4c79-9069-1eb8be07f869\") " pod="openshift-ovn-kubernetes/ovnkube-node-45dp8"
Sep 17 17:34:07.995338 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.994722    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/94a7bbd1-e769-4046-a56e-442208fb2ca2-etc-sysctl-conf\") pod \"tuned-n28f2\" (UID: \"94a7bbd1-e769-4046-a56e-442208fb2ca2\") " pod="openshift-cluster-node-tuning-operator/tuned-n28f2"
Sep 17 17:34:07.996235 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.994747    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bd527736-9920-4c79-9069-1eb8be07f869-host-cni-bin\") pod \"ovnkube-node-45dp8\" (UID: \"bd527736-9920-4c79-9069-1eb8be07f869\") " pod="openshift-ovn-kubernetes/ovnkube-node-45dp8"
Sep 17 17:34:07.996235 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.994749    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d7f1aae7-8961-4863-abff-c2259ef65ede-cnibin\") pod \"multus-additional-cni-plugins-g8nlc\" (UID: \"d7f1aae7-8961-4863-abff-c2259ef65ede\") " pod="openshift-multus/multus-additional-cni-plugins-g8nlc"
Sep 17 17:34:07.996235 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.994782    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bd527736-9920-4c79-9069-1eb8be07f869-host-slash\") pod \"ovnkube-node-45dp8\" (UID: \"bd527736-9920-4c79-9069-1eb8be07f869\") " pod="openshift-ovn-kubernetes/ovnkube-node-45dp8"
Sep 17 17:34:07.996235 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.994782    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d7f1aae7-8961-4863-abff-c2259ef65ede-cnibin\") pod \"multus-additional-cni-plugins-g8nlc\" (UID: \"d7f1aae7-8961-4863-abff-c2259ef65ede\") " pod="openshift-multus/multus-additional-cni-plugins-g8nlc"
Sep 17 17:34:07.996235 ip-10-0-134-96 kubenswrapper[2574]: E0917 17:34:07.994795    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c18eb096-7089-426a-957b-3dc1a2d7e837-metrics-certs podName:c18eb096-7089-426a-957b-3dc1a2d7e837 nodeName:}" failed. No retries permitted until 2026-09-17 17:34:08.494775316 +0000 UTC m=+3.683649005 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c18eb096-7089-426a-957b-3dc1a2d7e837-metrics-certs") pod "network-metrics-daemon-4tpm5" (UID: "c18eb096-7089-426a-957b-3dc1a2d7e837") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:34:07.996235 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.994830    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/bd527736-9920-4c79-9069-1eb8be07f869-host-cni-netd\") pod \"ovnkube-node-45dp8\" (UID: \"bd527736-9920-4c79-9069-1eb8be07f869\") " pod="openshift-ovn-kubernetes/ovnkube-node-45dp8"
Sep 17 17:34:07.996235 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.994840    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bd527736-9920-4c79-9069-1eb8be07f869-host-slash\") pod \"ovnkube-node-45dp8\" (UID: \"bd527736-9920-4c79-9069-1eb8be07f869\") " pod="openshift-ovn-kubernetes/ovnkube-node-45dp8"
Sep 17 17:34:07.996235 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.994857    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/bd527736-9920-4c79-9069-1eb8be07f869-env-overrides\") pod \"ovnkube-node-45dp8\" (UID: \"bd527736-9920-4c79-9069-1eb8be07f869\") " pod="openshift-ovn-kubernetes/ovnkube-node-45dp8"
Sep 17 17:34:07.996235 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.994879    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/fadd68fa-727c-4581-bed0-5ab8b73b9982-dbus\") pod \"global-pull-secret-syncer-sv2qq\" (UID: \"fadd68fa-727c-4581-bed0-5ab8b73b9982\") " pod="kube-system/global-pull-secret-syncer-sv2qq"
Sep 17 17:34:07.996235 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.994891    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d7f1aae7-8961-4863-abff-c2259ef65ede-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-g8nlc\" (UID: \"d7f1aae7-8961-4863-abff-c2259ef65ede\") " pod="openshift-multus/multus-additional-cni-plugins-g8nlc"
Sep 17 17:34:07.996235 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.994918    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/67b5a7ce-5a80-4224-9f54-86ac40fa136b-host\") pod \"node-ca-t79st\" (UID: \"67b5a7ce-5a80-4224-9f54-86ac40fa136b\") " pod="openshift-image-registry/node-ca-t79st"
Sep 17 17:34:07.996235 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.994882    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f33e242b-1311-43e7-8a5f-a97ad057236f-hostroot\") pod \"multus-ck426\" (UID: \"f33e242b-1311-43e7-8a5f-a97ad057236f\") " pod="openshift-multus/multus-ck426"
Sep 17 17:34:07.996235 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.994955    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f33e242b-1311-43e7-8a5f-a97ad057236f-hostroot\") pod \"multus-ck426\" (UID: \"f33e242b-1311-43e7-8a5f-a97ad057236f\") " pod="openshift-multus/multus-ck426"
Sep 17 17:34:07.996235 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.994918    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d7f1aae7-8961-4863-abff-c2259ef65ede-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-g8nlc\" (UID: \"d7f1aae7-8961-4863-abff-c2259ef65ede\") " pod="openshift-multus/multus-additional-cni-plugins-g8nlc"
Sep 17 17:34:07.996235 ip-10-0-134-96 kubenswrapper[2574]: E0917 17:34:07.994970    2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 17:34:07.996235 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.994955    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f33e242b-1311-43e7-8a5f-a97ad057236f-multus-conf-dir\") pod \"multus-ck426\" (UID: \"f33e242b-1311-43e7-8a5f-a97ad057236f\") " pod="openshift-multus/multus-ck426"
Sep 17 17:34:07.996235 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.994920    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/bd527736-9920-4c79-9069-1eb8be07f869-host-cni-netd\") pod \"ovnkube-node-45dp8\" (UID: \"bd527736-9920-4c79-9069-1eb8be07f869\") " pod="openshift-ovn-kubernetes/ovnkube-node-45dp8"
Sep 17 17:34:07.996963 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.995003    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d7f1aae7-8961-4863-abff-c2259ef65ede-tuning-conf-dir\") pod \"multus-additional-cni-plugins-g8nlc\" (UID: \"d7f1aae7-8961-4863-abff-c2259ef65ede\") " pod="openshift-multus/multus-additional-cni-plugins-g8nlc"
Sep 17 17:34:07.996963 ip-10-0-134-96 kubenswrapper[2574]: E0917 17:34:07.995024    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fadd68fa-727c-4581-bed0-5ab8b73b9982-original-pull-secret podName:fadd68fa-727c-4581-bed0-5ab8b73b9982 nodeName:}" failed. No retries permitted until 2026-09-17 17:34:08.495010299 +0000 UTC m=+3.683883990 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fadd68fa-727c-4581-bed0-5ab8b73b9982-original-pull-secret") pod "global-pull-secret-syncer-sv2qq" (UID: "fadd68fa-727c-4581-bed0-5ab8b73b9982") : object "kube-system"/"original-pull-secret" not registered
Sep 17 17:34:07.996963 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.995070    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/bd527736-9920-4c79-9069-1eb8be07f869-systemd-units\") pod \"ovnkube-node-45dp8\" (UID: \"bd527736-9920-4c79-9069-1eb8be07f869\") " pod="openshift-ovn-kubernetes/ovnkube-node-45dp8"
Sep 17 17:34:07.996963 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.995076    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f33e242b-1311-43e7-8a5f-a97ad057236f-multus-conf-dir\") pod \"multus-ck426\" (UID: \"f33e242b-1311-43e7-8a5f-a97ad057236f\") " pod="openshift-multus/multus-ck426"
Sep 17 17:34:07.996963 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.995077    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/94a7bbd1-e769-4046-a56e-442208fb2ca2-etc-sysctl-conf\") pod \"tuned-n28f2\" (UID: \"94a7bbd1-e769-4046-a56e-442208fb2ca2\") " pod="openshift-cluster-node-tuning-operator/tuned-n28f2"
Sep 17 17:34:07.996963 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.995094    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bd527736-9920-4c79-9069-1eb8be07f869-var-lib-openvswitch\") pod \"ovnkube-node-45dp8\" (UID: \"bd527736-9920-4c79-9069-1eb8be07f869\") " pod="openshift-ovn-kubernetes/ovnkube-node-45dp8"
Sep 17 17:34:07.996963 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.995120    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ff8b66be-df38-4e3f-9080-483b295fa79c-device-dir\") pod \"aws-ebs-csi-driver-node-rp7jd\" (UID: \"ff8b66be-df38-4e3f-9080-483b295fa79c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rp7jd"
Sep 17 17:34:07.996963 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.995122    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/bd527736-9920-4c79-9069-1eb8be07f869-systemd-units\") pod \"ovnkube-node-45dp8\" (UID: \"bd527736-9920-4c79-9069-1eb8be07f869\") " pod="openshift-ovn-kubernetes/ovnkube-node-45dp8"
Sep 17 17:34:07.996963 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.995131    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d7f1aae7-8961-4863-abff-c2259ef65ede-tuning-conf-dir\") pod \"multus-additional-cni-plugins-g8nlc\" (UID: \"d7f1aae7-8961-4863-abff-c2259ef65ede\") " pod="openshift-multus/multus-additional-cni-plugins-g8nlc"
Sep 17 17:34:07.996963 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.995144    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tgm22\" (UniqueName: \"kubernetes.io/projected/ff8b66be-df38-4e3f-9080-483b295fa79c-kube-api-access-tgm22\") pod \"aws-ebs-csi-driver-node-rp7jd\" (UID: \"ff8b66be-df38-4e3f-9080-483b295fa79c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rp7jd"
Sep 17 17:34:07.996963 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.995148    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bd527736-9920-4c79-9069-1eb8be07f869-var-lib-openvswitch\") pod \"ovnkube-node-45dp8\" (UID: \"bd527736-9920-4c79-9069-1eb8be07f869\") " pod="openshift-ovn-kubernetes/ovnkube-node-45dp8"
Sep 17 17:34:07.996963 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.995170    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/94a7bbd1-e769-4046-a56e-442208fb2ca2-etc-tuned\") pod \"tuned-n28f2\" (UID: \"94a7bbd1-e769-4046-a56e-442208fb2ca2\") " pod="openshift-cluster-node-tuning-operator/tuned-n28f2"
Sep 17 17:34:07.996963 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.995179    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ff8b66be-df38-4e3f-9080-483b295fa79c-device-dir\") pod \"aws-ebs-csi-driver-node-rp7jd\" (UID: \"ff8b66be-df38-4e3f-9080-483b295fa79c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rp7jd"
Sep 17 17:34:07.996963 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.995195    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d7f1aae7-8961-4863-abff-c2259ef65ede-cni-binary-copy\") pod \"multus-additional-cni-plugins-g8nlc\" (UID: \"d7f1aae7-8961-4863-abff-c2259ef65ede\") " pod="openshift-multus/multus-additional-cni-plugins-g8nlc"
Sep 17 17:34:07.996963 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.995220    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/fadd68fa-727c-4581-bed0-5ab8b73b9982-kubelet-config\") pod \"global-pull-secret-syncer-sv2qq\" (UID: \"fadd68fa-727c-4581-bed0-5ab8b73b9982\") " pod="kube-system/global-pull-secret-syncer-sv2qq"
Sep 17 17:34:07.996963 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.995243    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/94a7bbd1-e769-4046-a56e-442208fb2ca2-etc-kubernetes\") pod \"tuned-n28f2\" (UID: \"94a7bbd1-e769-4046-a56e-442208fb2ca2\") " pod="openshift-cluster-node-tuning-operator/tuned-n28f2"
Sep 17 17:34:07.997681 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.995264    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/94a7bbd1-e769-4046-a56e-442208fb2ca2-etc-sysctl-d\") pod \"tuned-n28f2\" (UID: \"94a7bbd1-e769-4046-a56e-442208fb2ca2\") " pod="openshift-cluster-node-tuning-operator/tuned-n28f2"
Sep 17 17:34:07.997681 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.995285    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/94a7bbd1-e769-4046-a56e-442208fb2ca2-tmp\") pod \"tuned-n28f2\" (UID: \"94a7bbd1-e769-4046-a56e-442208fb2ca2\") " pod="openshift-cluster-node-tuning-operator/tuned-n28f2"
Sep 17 17:34:07.997681 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.995308    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bd527736-9920-4c79-9069-1eb8be07f869-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-45dp8\" (UID: \"bd527736-9920-4c79-9069-1eb8be07f869\") " pod="openshift-ovn-kubernetes/ovnkube-node-45dp8"
Sep 17 17:34:07.997681 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.995323    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/fadd68fa-727c-4581-bed0-5ab8b73b9982-kubelet-config\") pod \"global-pull-secret-syncer-sv2qq\" (UID: \"fadd68fa-727c-4581-bed0-5ab8b73b9982\") " pod="kube-system/global-pull-secret-syncer-sv2qq"
Sep 17 17:34:07.997681 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.995334    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bd527736-9920-4c79-9069-1eb8be07f869-host-run-netns\") pod \"ovnkube-node-45dp8\" (UID: \"bd527736-9920-4c79-9069-1eb8be07f869\") " pod="openshift-ovn-kubernetes/ovnkube-node-45dp8"
Sep 17 17:34:07.997681 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.995338    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/bd527736-9920-4c79-9069-1eb8be07f869-env-overrides\") pod \"ovnkube-node-45dp8\" (UID: \"bd527736-9920-4c79-9069-1eb8be07f869\") " pod="openshift-ovn-kubernetes/ovnkube-node-45dp8"
Sep 17 17:34:07.997681 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.995383    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bd527736-9920-4c79-9069-1eb8be07f869-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-45dp8\" (UID: \"bd527736-9920-4c79-9069-1eb8be07f869\") " pod="openshift-ovn-kubernetes/ovnkube-node-45dp8"
Sep 17 17:34:07.997681 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.995415    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bd527736-9920-4c79-9069-1eb8be07f869-host-run-netns\") pod \"ovnkube-node-45dp8\" (UID: \"bd527736-9920-4c79-9069-1eb8be07f869\") " pod="openshift-ovn-kubernetes/ovnkube-node-45dp8"
Sep 17 17:34:07.997681 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.995421    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/94a7bbd1-e769-4046-a56e-442208fb2ca2-etc-sysctl-d\") pod \"tuned-n28f2\" (UID: \"94a7bbd1-e769-4046-a56e-442208fb2ca2\") " pod="openshift-cluster-node-tuning-operator/tuned-n28f2"
Sep 17 17:34:07.997681 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.995413    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/94a7bbd1-e769-4046-a56e-442208fb2ca2-etc-kubernetes\") pod \"tuned-n28f2\" (UID: \"94a7bbd1-e769-4046-a56e-442208fb2ca2\") " pod="openshift-cluster-node-tuning-operator/tuned-n28f2"
Sep 17 17:34:07.997681 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.995441    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ff8b66be-df38-4e3f-9080-483b295fa79c-registration-dir\") pod \"aws-ebs-csi-driver-node-rp7jd\" (UID: \"ff8b66be-df38-4e3f-9080-483b295fa79c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rp7jd"
Sep 17 17:34:07.997681 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.995466    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f33e242b-1311-43e7-8a5f-a97ad057236f-system-cni-dir\") pod \"multus-ck426\" (UID: \"f33e242b-1311-43e7-8a5f-a97ad057236f\") " pod="openshift-multus/multus-ck426"
Sep 17 17:34:07.997681 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.995482    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ff8b66be-df38-4e3f-9080-483b295fa79c-registration-dir\") pod \"aws-ebs-csi-driver-node-rp7jd\" (UID: \"ff8b66be-df38-4e3f-9080-483b295fa79c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rp7jd"
Sep 17 17:34:07.997681 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.995490    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f33e242b-1311-43e7-8a5f-a97ad057236f-host-run-netns\") pod \"multus-ck426\" (UID: \"f33e242b-1311-43e7-8a5f-a97ad057236f\") " pod="openshift-multus/multus-ck426"
Sep 17 17:34:07.997681 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.995439    2574 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 17 17:34:07.997681 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.995513    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d3f553e8-b416-4d7d-bbef-991a0404b287-agent-certs\") pod \"konnectivity-agent-2j4ds\" (UID: \"d3f553e8-b416-4d7d-bbef-991a0404b287\") " pod="kube-system/konnectivity-agent-2j4ds"
Sep 17 17:34:07.997681 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.995532    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/94a7bbd1-e769-4046-a56e-442208fb2ca2-host\") pod \"tuned-n28f2\" (UID: \"94a7bbd1-e769-4046-a56e-442208fb2ca2\") " pod="openshift-cluster-node-tuning-operator/tuned-n28f2"
Sep 17 17:34:07.997681 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.995551    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bsq5x\" (UniqueName: \"kubernetes.io/projected/31495fae-ecfa-4263-a3db-702f75935118-kube-api-access-bsq5x\") pod \"node-resolver-lf4tf\" (UID: \"31495fae-ecfa-4263-a3db-702f75935118\") " pod="openshift-dns/node-resolver-lf4tf"
Sep 17 17:34:07.998427 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.995560    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f33e242b-1311-43e7-8a5f-a97ad057236f-system-cni-dir\") pod \"multus-ck426\" (UID: \"f33e242b-1311-43e7-8a5f-a97ad057236f\") " pod="openshift-multus/multus-ck426"
Sep 17 17:34:07.998427 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.995572    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-894mr\" (UniqueName: \"kubernetes.io/projected/d7f1aae7-8961-4863-abff-c2259ef65ede-kube-api-access-894mr\") pod \"multus-additional-cni-plugins-g8nlc\" (UID: \"d7f1aae7-8961-4863-abff-c2259ef65ede\") " pod="openshift-multus/multus-additional-cni-plugins-g8nlc"
Sep 17 17:34:07.998427 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.995593    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/94a7bbd1-e769-4046-a56e-442208fb2ca2-var-lib-kubelet\") pod \"tuned-n28f2\" (UID: \"94a7bbd1-e769-4046-a56e-442208fb2ca2\") " pod="openshift-cluster-node-tuning-operator/tuned-n28f2"
Sep 17 17:34:07.998427 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.995551    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f33e242b-1311-43e7-8a5f-a97ad057236f-host-run-netns\") pod \"multus-ck426\" (UID: \"f33e242b-1311-43e7-8a5f-a97ad057236f\") " pod="openshift-multus/multus-ck426"
Sep 17 17:34:07.998427 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.995616    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-smzws\" (UniqueName: \"kubernetes.io/projected/bd527736-9920-4c79-9069-1eb8be07f869-kube-api-access-smzws\") pod \"ovnkube-node-45dp8\" (UID: \"bd527736-9920-4c79-9069-1eb8be07f869\") " pod="openshift-ovn-kubernetes/ovnkube-node-45dp8"
Sep 17 17:34:07.998427 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.995638    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f33e242b-1311-43e7-8a5f-a97ad057236f-multus-cni-dir\") pod \"multus-ck426\" (UID: \"f33e242b-1311-43e7-8a5f-a97ad057236f\") " pod="openshift-multus/multus-ck426"
Sep 17 17:34:07.998427 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.995652    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/94a7bbd1-e769-4046-a56e-442208fb2ca2-host\") pod \"tuned-n28f2\" (UID: \"94a7bbd1-e769-4046-a56e-442208fb2ca2\") " pod="openshift-cluster-node-tuning-operator/tuned-n28f2"
Sep 17 17:34:07.998427 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.995660    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f33e242b-1311-43e7-8a5f-a97ad057236f-host-var-lib-kubelet\") pod \"multus-ck426\" (UID: \"f33e242b-1311-43e7-8a5f-a97ad057236f\") " pod="openshift-multus/multus-ck426"
Sep 17 17:34:07.998427 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.995686    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/12264c60-6721-4820-816f-798eefa04bd5-iptables-alerter-script\") pod \"iptables-alerter-xkj8c\" (UID: \"12264c60-6721-4820-816f-798eefa04bd5\") " pod="openshift-network-operator/iptables-alerter-xkj8c"
Sep 17 17:34:07.998427 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.995712    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mbbqs\" (UniqueName: \"kubernetes.io/projected/12264c60-6721-4820-816f-798eefa04bd5-kube-api-access-mbbqs\") pod \"iptables-alerter-xkj8c\" (UID: \"12264c60-6721-4820-816f-798eefa04bd5\") " pod="openshift-network-operator/iptables-alerter-xkj8c"
Sep 17 17:34:07.998427 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.995736    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/94a7bbd1-e769-4046-a56e-442208fb2ca2-etc-modprobe-d\") pod \"tuned-n28f2\" (UID: \"94a7bbd1-e769-4046-a56e-442208fb2ca2\") " pod="openshift-cluster-node-tuning-operator/tuned-n28f2"
Sep 17 17:34:07.998427 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.995727    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d7f1aae7-8961-4863-abff-c2259ef65ede-cni-binary-copy\") pod \"multus-additional-cni-plugins-g8nlc\" (UID: \"d7f1aae7-8961-4863-abff-c2259ef65ede\") " pod="openshift-multus/multus-additional-cni-plugins-g8nlc"
Sep 17 17:34:07.998427 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.995752    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f33e242b-1311-43e7-8a5f-a97ad057236f-host-var-lib-kubelet\") pod \"multus-ck426\" (UID: \"f33e242b-1311-43e7-8a5f-a97ad057236f\") " pod="openshift-multus/multus-ck426"
Sep 17 17:34:07.998427 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.995758    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/94a7bbd1-e769-4046-a56e-442208fb2ca2-etc-sysconfig\") pod \"tuned-n28f2\" (UID: \"94a7bbd1-e769-4046-a56e-442208fb2ca2\") " pod="openshift-cluster-node-tuning-operator/tuned-n28f2"
Sep 17 17:34:07.998427 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.995797    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/bd527736-9920-4c79-9069-1eb8be07f869-log-socket\") pod \"ovnkube-node-45dp8\" (UID: \"bd527736-9920-4c79-9069-1eb8be07f869\") " pod="openshift-ovn-kubernetes/ovnkube-node-45dp8"
Sep 17 17:34:07.998427 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.995821    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/bd527736-9920-4c79-9069-1eb8be07f869-ovnkube-script-lib\") pod \"ovnkube-node-45dp8\" (UID: \"bd527736-9920-4c79-9069-1eb8be07f869\") " pod="openshift-ovn-kubernetes/ovnkube-node-45dp8"
Sep 17 17:34:07.998427 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.995828    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f33e242b-1311-43e7-8a5f-a97ad057236f-multus-cni-dir\") pod \"multus-ck426\" (UID: \"f33e242b-1311-43e7-8a5f-a97ad057236f\") " pod="openshift-multus/multus-ck426"
Sep 17 17:34:07.999666 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.995846    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f33e242b-1311-43e7-8a5f-a97ad057236f-multus-socket-dir-parent\") pod \"multus-ck426\" (UID: \"f33e242b-1311-43e7-8a5f-a97ad057236f\") " pod="openshift-multus/multus-ck426"
Sep 17 17:34:07.999666 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.995871    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f33e242b-1311-43e7-8a5f-a97ad057236f-host-run-multus-certs\") pod \"multus-ck426\" (UID: \"f33e242b-1311-43e7-8a5f-a97ad057236f\") " pod="openshift-multus/multus-ck426"
Sep 17 17:34:07.999666 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.995882    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/94a7bbd1-e769-4046-a56e-442208fb2ca2-etc-sysconfig\") pod \"tuned-n28f2\" (UID: \"94a7bbd1-e769-4046-a56e-442208fb2ca2\") " pod="openshift-cluster-node-tuning-operator/tuned-n28f2"
Sep 17 17:34:07.999666 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.995910    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/94a7bbd1-e769-4046-a56e-442208fb2ca2-var-lib-kubelet\") pod \"tuned-n28f2\" (UID: \"94a7bbd1-e769-4046-a56e-442208fb2ca2\") " pod="openshift-cluster-node-tuning-operator/tuned-n28f2"
Sep 17 17:34:07.999666 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.995895    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d3f553e8-b416-4d7d-bbef-991a0404b287-konnectivity-ca\") pod \"konnectivity-agent-2j4ds\" (UID: \"d3f553e8-b416-4d7d-bbef-991a0404b287\") " pod="kube-system/konnectivity-agent-2j4ds"
Sep 17 17:34:07.999666 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.995947    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/67b5a7ce-5a80-4224-9f54-86ac40fa136b-serviceca\") pod \"node-ca-t79st\" (UID: \"67b5a7ce-5a80-4224-9f54-86ac40fa136b\") " pod="openshift-image-registry/node-ca-t79st"
Sep 17 17:34:07.999666 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.995958    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/bd527736-9920-4c79-9069-1eb8be07f869-log-socket\") pod \"ovnkube-node-45dp8\" (UID: \"bd527736-9920-4c79-9069-1eb8be07f869\") " pod="openshift-ovn-kubernetes/ovnkube-node-45dp8"
Sep 17 17:34:07.999666 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.995972    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tp8c8\" (UniqueName: \"kubernetes.io/projected/67b5a7ce-5a80-4224-9f54-86ac40fa136b-kube-api-access-tp8c8\") pod \"node-ca-t79st\" (UID: \"67b5a7ce-5a80-4224-9f54-86ac40fa136b\") " pod="openshift-image-registry/node-ca-t79st"
Sep 17 17:34:07.999666 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.995980    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/94a7bbd1-e769-4046-a56e-442208fb2ca2-etc-modprobe-d\") pod \"tuned-n28f2\" (UID: \"94a7bbd1-e769-4046-a56e-442208fb2ca2\") " pod="openshift-cluster-node-tuning-operator/tuned-n28f2"
Sep 17 17:34:07.999666 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.995995    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f33e242b-1311-43e7-8a5f-a97ad057236f-host-run-multus-certs\") pod \"multus-ck426\" (UID: \"f33e242b-1311-43e7-8a5f-a97ad057236f\") " pod="openshift-multus/multus-ck426"
Sep 17 17:34:07.999666 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.995997    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vz647\" (UniqueName: \"kubernetes.io/projected/94a7bbd1-e769-4046-a56e-442208fb2ca2-kube-api-access-vz647\") pod \"tuned-n28f2\" (UID: \"94a7bbd1-e769-4046-a56e-442208fb2ca2\") " pod="openshift-cluster-node-tuning-operator/tuned-n28f2"
Sep 17 17:34:07.999666 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.996027    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f33e242b-1311-43e7-8a5f-a97ad057236f-multus-socket-dir-parent\") pod \"multus-ck426\" (UID: \"f33e242b-1311-43e7-8a5f-a97ad057236f\") " pod="openshift-multus/multus-ck426"
Sep 17 17:34:07.999666 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.996030    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/bd527736-9920-4c79-9069-1eb8be07f869-ovnkube-config\") pod \"ovnkube-node-45dp8\" (UID: \"bd527736-9920-4c79-9069-1eb8be07f869\") " pod="openshift-ovn-kubernetes/ovnkube-node-45dp8"
Sep 17 17:34:07.999666 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.996085    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/bd527736-9920-4c79-9069-1eb8be07f869-ovn-node-metrics-cert\") pod \"ovnkube-node-45dp8\" (UID: \"bd527736-9920-4c79-9069-1eb8be07f869\") " pod="openshift-ovn-kubernetes/ovnkube-node-45dp8"
Sep 17 17:34:07.999666 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.996111    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f33e242b-1311-43e7-8a5f-a97ad057236f-host-run-k8s-cni-cncf-io\") pod \"multus-ck426\" (UID: \"f33e242b-1311-43e7-8a5f-a97ad057236f\") " pod="openshift-multus/multus-ck426"
Sep 17 17:34:07.999666 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.996597    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/12264c60-6721-4820-816f-798eefa04bd5-iptables-alerter-script\") pod \"iptables-alerter-xkj8c\" (UID: \"12264c60-6721-4820-816f-798eefa04bd5\") " pod="openshift-network-operator/iptables-alerter-xkj8c"
Sep 17 17:34:07.999666 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.996854    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/67b5a7ce-5a80-4224-9f54-86ac40fa136b-serviceca\") pod \"node-ca-t79st\" (UID: \"67b5a7ce-5a80-4224-9f54-86ac40fa136b\") " pod="openshift-image-registry/node-ca-t79st"
Sep 17 17:34:07.999666 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.997130    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d3f553e8-b416-4d7d-bbef-991a0404b287-konnectivity-ca\") pod \"konnectivity-agent-2j4ds\" (UID: \"d3f553e8-b416-4d7d-bbef-991a0404b287\") " pod="kube-system/konnectivity-agent-2j4ds"
Sep 17 17:34:08.000498 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.997223    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/bd527736-9920-4c79-9069-1eb8be07f869-ovnkube-config\") pod \"ovnkube-node-45dp8\" (UID: \"bd527736-9920-4c79-9069-1eb8be07f869\") " pod="openshift-ovn-kubernetes/ovnkube-node-45dp8"
Sep 17 17:34:08.000498 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.997289    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f33e242b-1311-43e7-8a5f-a97ad057236f-host-run-k8s-cni-cncf-io\") pod \"multus-ck426\" (UID: \"f33e242b-1311-43e7-8a5f-a97ad057236f\") " pod="openshift-multus/multus-ck426"
Sep 17 17:34:08.000498 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.997577    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/bd527736-9920-4c79-9069-1eb8be07f869-ovnkube-script-lib\") pod \"ovnkube-node-45dp8\" (UID: \"bd527736-9920-4c79-9069-1eb8be07f869\") " pod="openshift-ovn-kubernetes/ovnkube-node-45dp8"
Sep 17 17:34:08.000498 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.999273    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/bd527736-9920-4c79-9069-1eb8be07f869-ovn-node-metrics-cert\") pod \"ovnkube-node-45dp8\" (UID: \"bd527736-9920-4c79-9069-1eb8be07f869\") " pod="openshift-ovn-kubernetes/ovnkube-node-45dp8"
Sep 17 17:34:08.000498 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:07.999391    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d3f553e8-b416-4d7d-bbef-991a0404b287-agent-certs\") pod \"konnectivity-agent-2j4ds\" (UID: \"d3f553e8-b416-4d7d-bbef-991a0404b287\") " pod="kube-system/konnectivity-agent-2j4ds"
Sep 17 17:34:08.000498 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:08.000180    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/94a7bbd1-e769-4046-a56e-442208fb2ca2-etc-tuned\") pod \"tuned-n28f2\" (UID: \"94a7bbd1-e769-4046-a56e-442208fb2ca2\") " pod="openshift-cluster-node-tuning-operator/tuned-n28f2"
Sep 17 17:34:08.001531 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:08.001510    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/94a7bbd1-e769-4046-a56e-442208fb2ca2-tmp\") pod \"tuned-n28f2\" (UID: \"94a7bbd1-e769-4046-a56e-442208fb2ca2\") " pod="openshift-cluster-node-tuning-operator/tuned-n28f2"
Sep 17 17:34:08.008310 ip-10-0-134-96 kubenswrapper[2574]: E0917 17:34:08.008290    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 17:34:08.008413 ip-10-0-134-96 kubenswrapper[2574]: E0917 17:34:08.008314    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 17:34:08.008413 ip-10-0-134-96 kubenswrapper[2574]: E0917 17:34:08.008327    2574 projected.go:196] Error preparing data for projected volume kube-api-access-2lk56 for pod openshift-network-diagnostics/network-check-target-j9d2w: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:34:08.008413 ip-10-0-134-96 kubenswrapper[2574]: E0917 17:34:08.008380    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/61c43e2e-f36c-401d-a10c-ff5f88fab086-kube-api-access-2lk56 podName:61c43e2e-f36c-401d-a10c-ff5f88fab086 nodeName:}" failed. No retries permitted until 2026-09-17 17:34:08.508364889 +0000 UTC m=+3.697238575 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-2lk56" (UniqueName: "kubernetes.io/projected/61c43e2e-f36c-401d-a10c-ff5f88fab086-kube-api-access-2lk56") pod "network-check-target-j9d2w" (UID: "61c43e2e-f36c-401d-a10c-ff5f88fab086") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:34:08.010792 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:08.010768    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mbbqs\" (UniqueName: \"kubernetes.io/projected/12264c60-6721-4820-816f-798eefa04bd5-kube-api-access-mbbqs\") pod \"iptables-alerter-xkj8c\" (UID: \"12264c60-6721-4820-816f-798eefa04bd5\") " pod="openshift-network-operator/iptables-alerter-xkj8c"
Sep 17 17:34:08.011239 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:08.011216    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tp8c8\" (UniqueName: \"kubernetes.io/projected/67b5a7ce-5a80-4224-9f54-86ac40fa136b-kube-api-access-tp8c8\") pod \"node-ca-t79st\" (UID: \"67b5a7ce-5a80-4224-9f54-86ac40fa136b\") " pod="openshift-image-registry/node-ca-t79st"
Sep 17 17:34:08.011336 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:08.011260    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bsq5x\" (UniqueName: \"kubernetes.io/projected/31495fae-ecfa-4263-a3db-702f75935118-kube-api-access-bsq5x\") pod \"node-resolver-lf4tf\" (UID: \"31495fae-ecfa-4263-a3db-702f75935118\") " pod="openshift-dns/node-resolver-lf4tf"
Sep 17 17:34:08.011516 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:08.011486    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tgm22\" (UniqueName: \"kubernetes.io/projected/ff8b66be-df38-4e3f-9080-483b295fa79c-kube-api-access-tgm22\") pod \"aws-ebs-csi-driver-node-rp7jd\" (UID: \"ff8b66be-df38-4e3f-9080-483b295fa79c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rp7jd"
Sep 17 17:34:08.011593 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:08.011582    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mk2pz\" (UniqueName: \"kubernetes.io/projected/c18eb096-7089-426a-957b-3dc1a2d7e837-kube-api-access-mk2pz\") pod \"network-metrics-daemon-4tpm5\" (UID: \"c18eb096-7089-426a-957b-3dc1a2d7e837\") " pod="openshift-multus/network-metrics-daemon-4tpm5"
Sep 17 17:34:08.011593 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:08.011582    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-smzws\" (UniqueName: \"kubernetes.io/projected/bd527736-9920-4c79-9069-1eb8be07f869-kube-api-access-smzws\") pod \"ovnkube-node-45dp8\" (UID: \"bd527736-9920-4c79-9069-1eb8be07f869\") " pod="openshift-ovn-kubernetes/ovnkube-node-45dp8"
Sep 17 17:34:08.011841 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:08.011824    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-894mr\" (UniqueName: \"kubernetes.io/projected/d7f1aae7-8961-4863-abff-c2259ef65ede-kube-api-access-894mr\") pod \"multus-additional-cni-plugins-g8nlc\" (UID: \"d7f1aae7-8961-4863-abff-c2259ef65ede\") " pod="openshift-multus/multus-additional-cni-plugins-g8nlc"
Sep 17 17:34:08.011901 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:08.011848    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vz647\" (UniqueName: \"kubernetes.io/projected/94a7bbd1-e769-4046-a56e-442208fb2ca2-kube-api-access-vz647\") pod \"tuned-n28f2\" (UID: \"94a7bbd1-e769-4046-a56e-442208fb2ca2\") " pod="openshift-cluster-node-tuning-operator/tuned-n28f2"
Sep 17 17:34:08.012368 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:08.012346    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h26rj\" (UniqueName: \"kubernetes.io/projected/f33e242b-1311-43e7-8a5f-a97ad057236f-kube-api-access-h26rj\") pod \"multus-ck426\" (UID: \"f33e242b-1311-43e7-8a5f-a97ad057236f\") " pod="openshift-multus/multus-ck426"
Sep 17 17:34:08.027397 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:08.027374    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ecb72cbea487238ef07d1e2c3698290.slice/crio-dba1ab8950e57cb917e6802a53b0baf7a24e932a60a4e8c1c395bd4ac298a6f9 WatchSource:0}: Error finding container dba1ab8950e57cb917e6802a53b0baf7a24e932a60a4e8c1c395bd4ac298a6f9: Status 404 returned error can't find the container with id dba1ab8950e57cb917e6802a53b0baf7a24e932a60a4e8c1c395bd4ac298a6f9
Sep 17 17:34:08.027793 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:08.027776    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee6fb7c1eb5fa5842b2aa3f6f4ac1869.slice/crio-3dd5750cca1f4511a6eb1d0abd69c756e4961ddb19a9b85e90a6501d6ae59bd5 WatchSource:0}: Error finding container 3dd5750cca1f4511a6eb1d0abd69c756e4961ddb19a9b85e90a6501d6ae59bd5: Status 404 returned error can't find the container with id 3dd5750cca1f4511a6eb1d0abd69c756e4961ddb19a9b85e90a6501d6ae59bd5
Sep 17 17:34:08.032727 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:08.032713    2574 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 17:34:08.048659 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:08.048640    2574 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 17:34:08.132319 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:08.132254    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-t79st"
Sep 17 17:34:08.141510 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:08.141486    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67b5a7ce_5a80_4224_9f54_86ac40fa136b.slice/crio-22c5afc11276cc8249c25fd70dc9fefab98762f4d5ce28a33bd89ea996e55991 WatchSource:0}: Error finding container 22c5afc11276cc8249c25fd70dc9fefab98762f4d5ce28a33bd89ea996e55991: Status 404 returned error can't find the container with id 22c5afc11276cc8249c25fd70dc9fefab98762f4d5ce28a33bd89ea996e55991
Sep 17 17:34:08.149547 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:08.149532    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-xkj8c"
Sep 17 17:34:08.156859 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:08.156838    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12264c60_6721_4820_816f_798eefa04bd5.slice/crio-f006680c1c14aad716ee39f255e2607399b4d8712d61139cf6965ac7ab0097a7 WatchSource:0}: Error finding container f006680c1c14aad716ee39f255e2607399b4d8712d61139cf6965ac7ab0097a7: Status 404 returned error can't find the container with id f006680c1c14aad716ee39f255e2607399b4d8712d61139cf6965ac7ab0097a7
Sep 17 17:34:08.169458 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:08.169441    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rp7jd"
Sep 17 17:34:08.177100 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:08.177080    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff8b66be_df38_4e3f_9080_483b295fa79c.slice/crio-11d2da5dbd643b6d828be18edc8ea6745cd8e08cdd9eff0be3bc0256ec6ae71f WatchSource:0}: Error finding container 11d2da5dbd643b6d828be18edc8ea6745cd8e08cdd9eff0be3bc0256ec6ae71f: Status 404 returned error can't find the container with id 11d2da5dbd643b6d828be18edc8ea6745cd8e08cdd9eff0be3bc0256ec6ae71f
Sep 17 17:34:08.186651 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:08.186636    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-ck426"
Sep 17 17:34:08.193349 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:08.193331    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-2j4ds"
Sep 17 17:34:08.194867 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:08.194850    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf33e242b_1311_43e7_8a5f_a97ad057236f.slice/crio-6a5a9ddc1e8c63049f05086b36baf5b579a58ac1f645421275e3b54c3b8da7e1 WatchSource:0}: Error finding container 6a5a9ddc1e8c63049f05086b36baf5b579a58ac1f645421275e3b54c3b8da7e1: Status 404 returned error can't find the container with id 6a5a9ddc1e8c63049f05086b36baf5b579a58ac1f645421275e3b54c3b8da7e1
Sep 17 17:34:08.201446 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:08.201426    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3f553e8_b416_4d7d_bbef_991a0404b287.slice/crio-248a497447b1405ba860de78622ad6c4bcb89a57c412fd62ac8198e56577e336 WatchSource:0}: Error finding container 248a497447b1405ba860de78622ad6c4bcb89a57c412fd62ac8198e56577e336: Status 404 returned error can't find the container with id 248a497447b1405ba860de78622ad6c4bcb89a57c412fd62ac8198e56577e336
Sep 17 17:34:08.223006 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:08.222988    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-n28f2"
Sep 17 17:34:08.230257 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:08.230237    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94a7bbd1_e769_4046_a56e_442208fb2ca2.slice/crio-e206ea5ce26f44b8bfe6e038552241be6ce5b681a147824a7a820fff6fc492a5 WatchSource:0}: Error finding container e206ea5ce26f44b8bfe6e038552241be6ce5b681a147824a7a820fff6fc492a5: Status 404 returned error can't find the container with id e206ea5ce26f44b8bfe6e038552241be6ce5b681a147824a7a820fff6fc492a5
Sep 17 17:34:08.237145 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:08.237085    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-lf4tf"
Sep 17 17:34:08.244642 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:08.244625    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31495fae_ecfa_4263_a3db_702f75935118.slice/crio-0bef83c946e92a60f20db9cde2ce8420a277947593b1d7698d405a638ee99a16 WatchSource:0}: Error finding container 0bef83c946e92a60f20db9cde2ce8420a277947593b1d7698d405a638ee99a16: Status 404 returned error can't find the container with id 0bef83c946e92a60f20db9cde2ce8420a277947593b1d7698d405a638ee99a16
Sep 17 17:34:08.254919 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:08.254905    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-g8nlc"
Sep 17 17:34:08.259469 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:08.259453    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-45dp8"
Sep 17 17:34:08.263336 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:08.263319    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7f1aae7_8961_4863_abff_c2259ef65ede.slice/crio-b39c5edb2f9c26699085090e824eccc34da3f8e29a6e5d7888328757dae806de WatchSource:0}: Error finding container b39c5edb2f9c26699085090e824eccc34da3f8e29a6e5d7888328757dae806de: Status 404 returned error can't find the container with id b39c5edb2f9c26699085090e824eccc34da3f8e29a6e5d7888328757dae806de
Sep 17 17:34:08.267320 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:08.267302    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd527736_9920_4c79_9069_1eb8be07f869.slice/crio-f6446890e6557f0e90640947392fe0a5494e1491793eae29b4beb46925a1efc4 WatchSource:0}: Error finding container f6446890e6557f0e90640947392fe0a5494e1491793eae29b4beb46925a1efc4: Status 404 returned error can't find the container with id f6446890e6557f0e90640947392fe0a5494e1491793eae29b4beb46925a1efc4
Sep 17 17:34:08.500754 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:08.500667    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fadd68fa-727c-4581-bed0-5ab8b73b9982-original-pull-secret\") pod \"global-pull-secret-syncer-sv2qq\" (UID: \"fadd68fa-727c-4581-bed0-5ab8b73b9982\") " pod="kube-system/global-pull-secret-syncer-sv2qq"
Sep 17 17:34:08.500754 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:08.500718    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c18eb096-7089-426a-957b-3dc1a2d7e837-metrics-certs\") pod \"network-metrics-daemon-4tpm5\" (UID: \"c18eb096-7089-426a-957b-3dc1a2d7e837\") " pod="openshift-multus/network-metrics-daemon-4tpm5"
Sep 17 17:34:08.500989 ip-10-0-134-96 kubenswrapper[2574]: E0917 17:34:08.500863    2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:34:08.501068 ip-10-0-134-96 kubenswrapper[2574]: E0917 17:34:08.501034    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c18eb096-7089-426a-957b-3dc1a2d7e837-metrics-certs podName:c18eb096-7089-426a-957b-3dc1a2d7e837 nodeName:}" failed. No retries permitted until 2026-09-17 17:34:09.501014991 +0000 UTC m=+4.689888677 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c18eb096-7089-426a-957b-3dc1a2d7e837-metrics-certs") pod "network-metrics-daemon-4tpm5" (UID: "c18eb096-7089-426a-957b-3dc1a2d7e837") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:34:08.501441 ip-10-0-134-96 kubenswrapper[2574]: E0917 17:34:08.500950    2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 17:34:08.501441 ip-10-0-134-96 kubenswrapper[2574]: E0917 17:34:08.501419    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fadd68fa-727c-4581-bed0-5ab8b73b9982-original-pull-secret podName:fadd68fa-727c-4581-bed0-5ab8b73b9982 nodeName:}" failed. No retries permitted until 2026-09-17 17:34:09.501404515 +0000 UTC m=+4.690278214 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fadd68fa-727c-4581-bed0-5ab8b73b9982-original-pull-secret") pod "global-pull-secret-syncer-sv2qq" (UID: "fadd68fa-727c-4581-bed0-5ab8b73b9982") : object "kube-system"/"original-pull-secret" not registered
Sep 17 17:34:08.602137 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:08.601467    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2lk56\" (UniqueName: \"kubernetes.io/projected/61c43e2e-f36c-401d-a10c-ff5f88fab086-kube-api-access-2lk56\") pod \"network-check-target-j9d2w\" (UID: \"61c43e2e-f36c-401d-a10c-ff5f88fab086\") " pod="openshift-network-diagnostics/network-check-target-j9d2w"
Sep 17 17:34:08.602137 ip-10-0-134-96 kubenswrapper[2574]: E0917 17:34:08.601675    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 17:34:08.602137 ip-10-0-134-96 kubenswrapper[2574]: E0917 17:34:08.601696    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 17:34:08.602137 ip-10-0-134-96 kubenswrapper[2574]: E0917 17:34:08.601710    2574 projected.go:196] Error preparing data for projected volume kube-api-access-2lk56 for pod openshift-network-diagnostics/network-check-target-j9d2w: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:34:08.602137 ip-10-0-134-96 kubenswrapper[2574]: E0917 17:34:08.601776    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/61c43e2e-f36c-401d-a10c-ff5f88fab086-kube-api-access-2lk56 podName:61c43e2e-f36c-401d-a10c-ff5f88fab086 nodeName:}" failed. No retries permitted until 2026-09-17 17:34:09.601758891 +0000 UTC m=+4.790632582 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-2lk56" (UniqueName: "kubernetes.io/projected/61c43e2e-f36c-401d-a10c-ff5f88fab086-kube-api-access-2lk56") pod "network-check-target-j9d2w" (UID: "61c43e2e-f36c-401d-a10c-ff5f88fab086") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:34:08.728061 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:08.727867    2574 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 17:34:08.847730 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:08.847608    2574 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-16 17:29:07 +0000 UTC" deadline="2028-02-18 03:09:40.218752856 +0000 UTC"
Sep 17 17:34:08.847730 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:08.847641    2574 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12441h35m31.37111611s"
Sep 17 17:34:08.968680 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:08.968621    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rp7jd" event={"ID":"ff8b66be-df38-4e3f-9080-483b295fa79c","Type":"ContainerStarted","Data":"11d2da5dbd643b6d828be18edc8ea6745cd8e08cdd9eff0be3bc0256ec6ae71f"}
Sep 17 17:34:08.991224 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:08.991148    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-xkj8c" event={"ID":"12264c60-6721-4820-816f-798eefa04bd5","Type":"ContainerStarted","Data":"f006680c1c14aad716ee39f255e2607399b4d8712d61139cf6965ac7ab0097a7"}
Sep 17 17:34:08.995676 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:08.995646    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-t79st" event={"ID":"67b5a7ce-5a80-4224-9f54-86ac40fa136b","Type":"ContainerStarted","Data":"22c5afc11276cc8249c25fd70dc9fefab98762f4d5ce28a33bd89ea996e55991"}
Sep 17 17:34:09.004228 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:09.004153    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-lf4tf" event={"ID":"31495fae-ecfa-4263-a3db-702f75935118","Type":"ContainerStarted","Data":"0bef83c946e92a60f20db9cde2ce8420a277947593b1d7698d405a638ee99a16"}
Sep 17 17:34:09.021018 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:09.020940    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-96.ec2.internal" event={"ID":"ee6fb7c1eb5fa5842b2aa3f6f4ac1869","Type":"ContainerStarted","Data":"3dd5750cca1f4511a6eb1d0abd69c756e4961ddb19a9b85e90a6501d6ae59bd5"}
Sep 17 17:34:09.025974 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:09.025921    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-96.ec2.internal" event={"ID":"5ecb72cbea487238ef07d1e2c3698290","Type":"ContainerStarted","Data":"dba1ab8950e57cb917e6802a53b0baf7a24e932a60a4e8c1c395bd4ac298a6f9"}
Sep 17 17:34:09.029982 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:09.029954    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-45dp8" event={"ID":"bd527736-9920-4c79-9069-1eb8be07f869","Type":"ContainerStarted","Data":"f6446890e6557f0e90640947392fe0a5494e1491793eae29b4beb46925a1efc4"}
Sep 17 17:34:09.047768 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:09.047697    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-g8nlc" event={"ID":"d7f1aae7-8961-4863-abff-c2259ef65ede","Type":"ContainerStarted","Data":"b39c5edb2f9c26699085090e824eccc34da3f8e29a6e5d7888328757dae806de"}
Sep 17 17:34:09.066839 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:09.066812    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-n28f2" event={"ID":"94a7bbd1-e769-4046-a56e-442208fb2ca2","Type":"ContainerStarted","Data":"e206ea5ce26f44b8bfe6e038552241be6ce5b681a147824a7a820fff6fc492a5"}
Sep 17 17:34:09.083688 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:09.083658    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-2j4ds" event={"ID":"d3f553e8-b416-4d7d-bbef-991a0404b287","Type":"ContainerStarted","Data":"248a497447b1405ba860de78622ad6c4bcb89a57c412fd62ac8198e56577e336"}
Sep 17 17:34:09.101982 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:09.101871    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-ck426" event={"ID":"f33e242b-1311-43e7-8a5f-a97ad057236f","Type":"ContainerStarted","Data":"6a5a9ddc1e8c63049f05086b36baf5b579a58ac1f645421275e3b54c3b8da7e1"}
Sep 17 17:34:09.420927 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:09.420852    2574 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 17:34:09.509070 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:09.508341    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fadd68fa-727c-4581-bed0-5ab8b73b9982-original-pull-secret\") pod \"global-pull-secret-syncer-sv2qq\" (UID: \"fadd68fa-727c-4581-bed0-5ab8b73b9982\") " pod="kube-system/global-pull-secret-syncer-sv2qq"
Sep 17 17:34:09.509070 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:09.508387    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c18eb096-7089-426a-957b-3dc1a2d7e837-metrics-certs\") pod \"network-metrics-daemon-4tpm5\" (UID: \"c18eb096-7089-426a-957b-3dc1a2d7e837\") " pod="openshift-multus/network-metrics-daemon-4tpm5"
Sep 17 17:34:09.509070 ip-10-0-134-96 kubenswrapper[2574]: E0917 17:34:09.508516    2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:34:09.509070 ip-10-0-134-96 kubenswrapper[2574]: E0917 17:34:09.508573    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c18eb096-7089-426a-957b-3dc1a2d7e837-metrics-certs podName:c18eb096-7089-426a-957b-3dc1a2d7e837 nodeName:}" failed. No retries permitted until 2026-09-17 17:34:11.508555408 +0000 UTC m=+6.697429099 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c18eb096-7089-426a-957b-3dc1a2d7e837-metrics-certs") pod "network-metrics-daemon-4tpm5" (UID: "c18eb096-7089-426a-957b-3dc1a2d7e837") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:34:09.509070 ip-10-0-134-96 kubenswrapper[2574]: E0917 17:34:09.508977    2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 17:34:09.509070 ip-10-0-134-96 kubenswrapper[2574]: E0917 17:34:09.509024    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fadd68fa-727c-4581-bed0-5ab8b73b9982-original-pull-secret podName:fadd68fa-727c-4581-bed0-5ab8b73b9982 nodeName:}" failed. No retries permitted until 2026-09-17 17:34:11.509010148 +0000 UTC m=+6.697883836 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fadd68fa-727c-4581-bed0-5ab8b73b9982-original-pull-secret") pod "global-pull-secret-syncer-sv2qq" (UID: "fadd68fa-727c-4581-bed0-5ab8b73b9982") : object "kube-system"/"original-pull-secret" not registered
Sep 17 17:34:09.609249 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:09.609218    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2lk56\" (UniqueName: \"kubernetes.io/projected/61c43e2e-f36c-401d-a10c-ff5f88fab086-kube-api-access-2lk56\") pod \"network-check-target-j9d2w\" (UID: \"61c43e2e-f36c-401d-a10c-ff5f88fab086\") " pod="openshift-network-diagnostics/network-check-target-j9d2w"
Sep 17 17:34:09.609452 ip-10-0-134-96 kubenswrapper[2574]: E0917 17:34:09.609422    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 17:34:09.609452 ip-10-0-134-96 kubenswrapper[2574]: E0917 17:34:09.609447    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 17:34:09.609592 ip-10-0-134-96 kubenswrapper[2574]: E0917 17:34:09.609460    2574 projected.go:196] Error preparing data for projected volume kube-api-access-2lk56 for pod openshift-network-diagnostics/network-check-target-j9d2w: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:34:09.609592 ip-10-0-134-96 kubenswrapper[2574]: E0917 17:34:09.609515    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/61c43e2e-f36c-401d-a10c-ff5f88fab086-kube-api-access-2lk56 podName:61c43e2e-f36c-401d-a10c-ff5f88fab086 nodeName:}" failed. No retries permitted until 2026-09-17 17:34:11.609497509 +0000 UTC m=+6.798371200 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-2lk56" (UniqueName: "kubernetes.io/projected/61c43e2e-f36c-401d-a10c-ff5f88fab086-kube-api-access-2lk56") pod "network-check-target-j9d2w" (UID: "61c43e2e-f36c-401d-a10c-ff5f88fab086") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:34:09.849010 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:09.848850    2574 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-16 17:29:07 +0000 UTC" deadline="2028-06-14 06:10:28.584811668 +0000 UTC"
Sep 17 17:34:09.849010 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:09.848958    2574 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15252h36m18.735857802s"
Sep 17 17:34:09.948279 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:09.948249    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4tpm5"
Sep 17 17:34:09.948465 ip-10-0-134-96 kubenswrapper[2574]: E0917 17:34:09.948389    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4tpm5" podUID="c18eb096-7089-426a-957b-3dc1a2d7e837"
Sep 17 17:34:09.948532 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:09.948476    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sv2qq"
Sep 17 17:34:09.948586 ip-10-0-134-96 kubenswrapper[2574]: E0917 17:34:09.948557    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sv2qq" podUID="fadd68fa-727c-4581-bed0-5ab8b73b9982"
Sep 17 17:34:09.948642 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:09.948623    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j9d2w"
Sep 17 17:34:09.948716 ip-10-0-134-96 kubenswrapper[2574]: E0917 17:34:09.948690    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j9d2w" podUID="61c43e2e-f36c-401d-a10c-ff5f88fab086"
Sep 17 17:34:10.105576 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:10.105502    2574 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 17:34:11.526075 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:11.526029    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fadd68fa-727c-4581-bed0-5ab8b73b9982-original-pull-secret\") pod \"global-pull-secret-syncer-sv2qq\" (UID: \"fadd68fa-727c-4581-bed0-5ab8b73b9982\") " pod="kube-system/global-pull-secret-syncer-sv2qq"
Sep 17 17:34:11.526562 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:11.526088    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c18eb096-7089-426a-957b-3dc1a2d7e837-metrics-certs\") pod \"network-metrics-daemon-4tpm5\" (UID: \"c18eb096-7089-426a-957b-3dc1a2d7e837\") " pod="openshift-multus/network-metrics-daemon-4tpm5"
Sep 17 17:34:11.526562 ip-10-0-134-96 kubenswrapper[2574]: E0917 17:34:11.526256    2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:34:11.526562 ip-10-0-134-96 kubenswrapper[2574]: E0917 17:34:11.526310    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c18eb096-7089-426a-957b-3dc1a2d7e837-metrics-certs podName:c18eb096-7089-426a-957b-3dc1a2d7e837 nodeName:}" failed. No retries permitted until 2026-09-17 17:34:15.526293128 +0000 UTC m=+10.715166820 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c18eb096-7089-426a-957b-3dc1a2d7e837-metrics-certs") pod "network-metrics-daemon-4tpm5" (UID: "c18eb096-7089-426a-957b-3dc1a2d7e837") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:34:11.526741 ip-10-0-134-96 kubenswrapper[2574]: E0917 17:34:11.526670    2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 17:34:11.526741 ip-10-0-134-96 kubenswrapper[2574]: E0917 17:34:11.526713    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fadd68fa-727c-4581-bed0-5ab8b73b9982-original-pull-secret podName:fadd68fa-727c-4581-bed0-5ab8b73b9982 nodeName:}" failed. No retries permitted until 2026-09-17 17:34:15.526698939 +0000 UTC m=+10.715572630 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fadd68fa-727c-4581-bed0-5ab8b73b9982-original-pull-secret") pod "global-pull-secret-syncer-sv2qq" (UID: "fadd68fa-727c-4581-bed0-5ab8b73b9982") : object "kube-system"/"original-pull-secret" not registered
Sep 17 17:34:11.627154 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:11.626551    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2lk56\" (UniqueName: \"kubernetes.io/projected/61c43e2e-f36c-401d-a10c-ff5f88fab086-kube-api-access-2lk56\") pod \"network-check-target-j9d2w\" (UID: \"61c43e2e-f36c-401d-a10c-ff5f88fab086\") " pod="openshift-network-diagnostics/network-check-target-j9d2w"
Sep 17 17:34:11.627154 ip-10-0-134-96 kubenswrapper[2574]: E0917 17:34:11.626724    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 17:34:11.627154 ip-10-0-134-96 kubenswrapper[2574]: E0917 17:34:11.626744    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 17:34:11.627154 ip-10-0-134-96 kubenswrapper[2574]: E0917 17:34:11.626757    2574 projected.go:196] Error preparing data for projected volume kube-api-access-2lk56 for pod openshift-network-diagnostics/network-check-target-j9d2w: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:34:11.627154 ip-10-0-134-96 kubenswrapper[2574]: E0917 17:34:11.626809    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/61c43e2e-f36c-401d-a10c-ff5f88fab086-kube-api-access-2lk56 podName:61c43e2e-f36c-401d-a10c-ff5f88fab086 nodeName:}" failed. No retries permitted until 2026-09-17 17:34:15.626791306 +0000 UTC m=+10.815664990 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-2lk56" (UniqueName: "kubernetes.io/projected/61c43e2e-f36c-401d-a10c-ff5f88fab086-kube-api-access-2lk56") pod "network-check-target-j9d2w" (UID: "61c43e2e-f36c-401d-a10c-ff5f88fab086") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:34:11.948850 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:11.948745    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4tpm5"
Sep 17 17:34:11.948850 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:11.948760    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sv2qq"
Sep 17 17:34:11.949077 ip-10-0-134-96 kubenswrapper[2574]: E0917 17:34:11.948868    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sv2qq" podUID="fadd68fa-727c-4581-bed0-5ab8b73b9982"
Sep 17 17:34:11.949430 ip-10-0-134-96 kubenswrapper[2574]: E0917 17:34:11.949251    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4tpm5" podUID="c18eb096-7089-426a-957b-3dc1a2d7e837"
Sep 17 17:34:11.949430 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:11.949318    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j9d2w"
Sep 17 17:34:11.949430 ip-10-0-134-96 kubenswrapper[2574]: E0917 17:34:11.949398    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j9d2w" podUID="61c43e2e-f36c-401d-a10c-ff5f88fab086"
Sep 17 17:34:13.948747 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:13.948675    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4tpm5"
Sep 17 17:34:13.948747 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:13.948703    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j9d2w"
Sep 17 17:34:13.949276 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:13.948692    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sv2qq"
Sep 17 17:34:13.949276 ip-10-0-134-96 kubenswrapper[2574]: E0917 17:34:13.948847    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4tpm5" podUID="c18eb096-7089-426a-957b-3dc1a2d7e837"
Sep 17 17:34:13.949276 ip-10-0-134-96 kubenswrapper[2574]: E0917 17:34:13.948946    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j9d2w" podUID="61c43e2e-f36c-401d-a10c-ff5f88fab086"
Sep 17 17:34:13.949433 ip-10-0-134-96 kubenswrapper[2574]: E0917 17:34:13.949352    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sv2qq" podUID="fadd68fa-727c-4581-bed0-5ab8b73b9982"
Sep 17 17:34:15.561908 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:15.561864    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fadd68fa-727c-4581-bed0-5ab8b73b9982-original-pull-secret\") pod \"global-pull-secret-syncer-sv2qq\" (UID: \"fadd68fa-727c-4581-bed0-5ab8b73b9982\") " pod="kube-system/global-pull-secret-syncer-sv2qq"
Sep 17 17:34:15.562368 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:15.561916    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c18eb096-7089-426a-957b-3dc1a2d7e837-metrics-certs\") pod \"network-metrics-daemon-4tpm5\" (UID: \"c18eb096-7089-426a-957b-3dc1a2d7e837\") " pod="openshift-multus/network-metrics-daemon-4tpm5"
Sep 17 17:34:15.562368 ip-10-0-134-96 kubenswrapper[2574]: E0917 17:34:15.562069    2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:34:15.562368 ip-10-0-134-96 kubenswrapper[2574]: E0917 17:34:15.562131    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c18eb096-7089-426a-957b-3dc1a2d7e837-metrics-certs podName:c18eb096-7089-426a-957b-3dc1a2d7e837 nodeName:}" failed. No retries permitted until 2026-09-17 17:34:23.562111137 +0000 UTC m=+18.750984826 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c18eb096-7089-426a-957b-3dc1a2d7e837-metrics-certs") pod "network-metrics-daemon-4tpm5" (UID: "c18eb096-7089-426a-957b-3dc1a2d7e837") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:34:15.562628 ip-10-0-134-96 kubenswrapper[2574]: E0917 17:34:15.562604    2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 17:34:15.562700 ip-10-0-134-96 kubenswrapper[2574]: E0917 17:34:15.562676    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fadd68fa-727c-4581-bed0-5ab8b73b9982-original-pull-secret podName:fadd68fa-727c-4581-bed0-5ab8b73b9982 nodeName:}" failed. No retries permitted until 2026-09-17 17:34:23.56265649 +0000 UTC m=+18.751530180 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fadd68fa-727c-4581-bed0-5ab8b73b9982-original-pull-secret") pod "global-pull-secret-syncer-sv2qq" (UID: "fadd68fa-727c-4581-bed0-5ab8b73b9982") : object "kube-system"/"original-pull-secret" not registered
Sep 17 17:34:15.662446 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:15.662235    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2lk56\" (UniqueName: \"kubernetes.io/projected/61c43e2e-f36c-401d-a10c-ff5f88fab086-kube-api-access-2lk56\") pod \"network-check-target-j9d2w\" (UID: \"61c43e2e-f36c-401d-a10c-ff5f88fab086\") " pod="openshift-network-diagnostics/network-check-target-j9d2w"
Sep 17 17:34:15.662621 ip-10-0-134-96 kubenswrapper[2574]: E0917 17:34:15.662456    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 17:34:15.662621 ip-10-0-134-96 kubenswrapper[2574]: E0917 17:34:15.662481    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 17:34:15.662621 ip-10-0-134-96 kubenswrapper[2574]: E0917 17:34:15.662494    2574 projected.go:196] Error preparing data for projected volume kube-api-access-2lk56 for pod openshift-network-diagnostics/network-check-target-j9d2w: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:34:15.662621 ip-10-0-134-96 kubenswrapper[2574]: E0917 17:34:15.662556    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/61c43e2e-f36c-401d-a10c-ff5f88fab086-kube-api-access-2lk56 podName:61c43e2e-f36c-401d-a10c-ff5f88fab086 nodeName:}" failed. No retries permitted until 2026-09-17 17:34:23.662537345 +0000 UTC m=+18.851411047 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-2lk56" (UniqueName: "kubernetes.io/projected/61c43e2e-f36c-401d-a10c-ff5f88fab086-kube-api-access-2lk56") pod "network-check-target-j9d2w" (UID: "61c43e2e-f36c-401d-a10c-ff5f88fab086") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:34:15.948334 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:15.948148    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4tpm5"
Sep 17 17:34:15.948334 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:15.948187    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sv2qq"
Sep 17 17:34:15.948334 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:15.948193    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j9d2w"
Sep 17 17:34:15.948334 ip-10-0-134-96 kubenswrapper[2574]: E0917 17:34:15.948289    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4tpm5" podUID="c18eb096-7089-426a-957b-3dc1a2d7e837"
Sep 17 17:34:15.948699 ip-10-0-134-96 kubenswrapper[2574]: E0917 17:34:15.948373    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sv2qq" podUID="fadd68fa-727c-4581-bed0-5ab8b73b9982"
Sep 17 17:34:15.948699 ip-10-0-134-96 kubenswrapper[2574]: E0917 17:34:15.948447    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j9d2w" podUID="61c43e2e-f36c-401d-a10c-ff5f88fab086"
Sep 17 17:34:17.948726 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:17.948690    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4tpm5"
Sep 17 17:34:17.949283 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:17.948737    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sv2qq"
Sep 17 17:34:17.949283 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:17.948817    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j9d2w"
Sep 17 17:34:17.949283 ip-10-0-134-96 kubenswrapper[2574]: E0917 17:34:17.948835    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4tpm5" podUID="c18eb096-7089-426a-957b-3dc1a2d7e837"
Sep 17 17:34:17.949283 ip-10-0-134-96 kubenswrapper[2574]: E0917 17:34:17.948903    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j9d2w" podUID="61c43e2e-f36c-401d-a10c-ff5f88fab086"
Sep 17 17:34:17.949283 ip-10-0-134-96 kubenswrapper[2574]: E0917 17:34:17.949055    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sv2qq" podUID="fadd68fa-727c-4581-bed0-5ab8b73b9982"
Sep 17 17:34:19.948832 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:19.948784    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j9d2w"
Sep 17 17:34:19.948832 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:19.948814    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sv2qq"
Sep 17 17:34:19.949383 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:19.948814    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4tpm5"
Sep 17 17:34:19.949383 ip-10-0-134-96 kubenswrapper[2574]: E0917 17:34:19.948909    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j9d2w" podUID="61c43e2e-f36c-401d-a10c-ff5f88fab086"
Sep 17 17:34:19.949383 ip-10-0-134-96 kubenswrapper[2574]: E0917 17:34:19.948993    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sv2qq" podUID="fadd68fa-727c-4581-bed0-5ab8b73b9982"
Sep 17 17:34:19.949383 ip-10-0-134-96 kubenswrapper[2574]: E0917 17:34:19.949090    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4tpm5" podUID="c18eb096-7089-426a-957b-3dc1a2d7e837"
Sep 17 17:34:21.948033 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:21.948002    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4tpm5"
Sep 17 17:34:21.948490 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:21.948144    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sv2qq"
Sep 17 17:34:21.948490 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:21.948154    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j9d2w"
Sep 17 17:34:21.948490 ip-10-0-134-96 kubenswrapper[2574]: E0917 17:34:21.948153    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4tpm5" podUID="c18eb096-7089-426a-957b-3dc1a2d7e837"
Sep 17 17:34:21.948490 ip-10-0-134-96 kubenswrapper[2574]: E0917 17:34:21.948256    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sv2qq" podUID="fadd68fa-727c-4581-bed0-5ab8b73b9982"
Sep 17 17:34:21.948490 ip-10-0-134-96 kubenswrapper[2574]: E0917 17:34:21.948326    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j9d2w" podUID="61c43e2e-f36c-401d-a10c-ff5f88fab086"
Sep 17 17:34:23.616034 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:23.615979    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fadd68fa-727c-4581-bed0-5ab8b73b9982-original-pull-secret\") pod \"global-pull-secret-syncer-sv2qq\" (UID: \"fadd68fa-727c-4581-bed0-5ab8b73b9982\") " pod="kube-system/global-pull-secret-syncer-sv2qq"
Sep 17 17:34:23.616504 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:23.616063    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c18eb096-7089-426a-957b-3dc1a2d7e837-metrics-certs\") pod \"network-metrics-daemon-4tpm5\" (UID: \"c18eb096-7089-426a-957b-3dc1a2d7e837\") " pod="openshift-multus/network-metrics-daemon-4tpm5"
Sep 17 17:34:23.616504 ip-10-0-134-96 kubenswrapper[2574]: E0917 17:34:23.616130    2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 17:34:23.616504 ip-10-0-134-96 kubenswrapper[2574]: E0917 17:34:23.616172    2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:34:23.616504 ip-10-0-134-96 kubenswrapper[2574]: E0917 17:34:23.616210    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fadd68fa-727c-4581-bed0-5ab8b73b9982-original-pull-secret podName:fadd68fa-727c-4581-bed0-5ab8b73b9982 nodeName:}" failed. No retries permitted until 2026-09-17 17:34:39.616189305 +0000 UTC m=+34.805062991 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fadd68fa-727c-4581-bed0-5ab8b73b9982-original-pull-secret") pod "global-pull-secret-syncer-sv2qq" (UID: "fadd68fa-727c-4581-bed0-5ab8b73b9982") : object "kube-system"/"original-pull-secret" not registered
Sep 17 17:34:23.616504 ip-10-0-134-96 kubenswrapper[2574]: E0917 17:34:23.616231    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c18eb096-7089-426a-957b-3dc1a2d7e837-metrics-certs podName:c18eb096-7089-426a-957b-3dc1a2d7e837 nodeName:}" failed. No retries permitted until 2026-09-17 17:34:39.616219477 +0000 UTC m=+34.805093162 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c18eb096-7089-426a-957b-3dc1a2d7e837-metrics-certs") pod "network-metrics-daemon-4tpm5" (UID: "c18eb096-7089-426a-957b-3dc1a2d7e837") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:34:23.716565 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:23.716528    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2lk56\" (UniqueName: \"kubernetes.io/projected/61c43e2e-f36c-401d-a10c-ff5f88fab086-kube-api-access-2lk56\") pod \"network-check-target-j9d2w\" (UID: \"61c43e2e-f36c-401d-a10c-ff5f88fab086\") " pod="openshift-network-diagnostics/network-check-target-j9d2w"
Sep 17 17:34:23.716749 ip-10-0-134-96 kubenswrapper[2574]: E0917 17:34:23.716719    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 17:34:23.716749 ip-10-0-134-96 kubenswrapper[2574]: E0917 17:34:23.716742    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 17:34:23.716846 ip-10-0-134-96 kubenswrapper[2574]: E0917 17:34:23.716752    2574 projected.go:196] Error preparing data for projected volume kube-api-access-2lk56 for pod openshift-network-diagnostics/network-check-target-j9d2w: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:34:23.716846 ip-10-0-134-96 kubenswrapper[2574]: E0917 17:34:23.716814    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/61c43e2e-f36c-401d-a10c-ff5f88fab086-kube-api-access-2lk56 podName:61c43e2e-f36c-401d-a10c-ff5f88fab086 nodeName:}" failed. No retries permitted until 2026-09-17 17:34:39.716794603 +0000 UTC m=+34.905668287 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-2lk56" (UniqueName: "kubernetes.io/projected/61c43e2e-f36c-401d-a10c-ff5f88fab086-kube-api-access-2lk56") pod "network-check-target-j9d2w" (UID: "61c43e2e-f36c-401d-a10c-ff5f88fab086") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:34:23.948163 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:23.948075    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4tpm5"
Sep 17 17:34:23.948163 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:23.948113    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sv2qq"
Sep 17 17:34:23.948163 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:23.948119    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j9d2w"
Sep 17 17:34:23.948424 ip-10-0-134-96 kubenswrapper[2574]: E0917 17:34:23.948251    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4tpm5" podUID="c18eb096-7089-426a-957b-3dc1a2d7e837"
Sep 17 17:34:23.948424 ip-10-0-134-96 kubenswrapper[2574]: E0917 17:34:23.948322    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sv2qq" podUID="fadd68fa-727c-4581-bed0-5ab8b73b9982"
Sep 17 17:34:23.948424 ip-10-0-134-96 kubenswrapper[2574]: E0917 17:34:23.948401    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j9d2w" podUID="61c43e2e-f36c-401d-a10c-ff5f88fab086"
Sep 17 17:34:25.948397 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:25.948352    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sv2qq"
Sep 17 17:34:25.948397 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:25.948376    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4tpm5"
Sep 17 17:34:25.948397 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:25.948402    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j9d2w"
Sep 17 17:34:25.948951 ip-10-0-134-96 kubenswrapper[2574]: E0917 17:34:25.948507    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sv2qq" podUID="fadd68fa-727c-4581-bed0-5ab8b73b9982"
Sep 17 17:34:25.948951 ip-10-0-134-96 kubenswrapper[2574]: E0917 17:34:25.948587    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4tpm5" podUID="c18eb096-7089-426a-957b-3dc1a2d7e837"
Sep 17 17:34:25.948951 ip-10-0-134-96 kubenswrapper[2574]: E0917 17:34:25.948663    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j9d2w" podUID="61c43e2e-f36c-401d-a10c-ff5f88fab086"
Sep 17 17:34:27.146521 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:27.146081    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-96.ec2.internal" event={"ID":"ee6fb7c1eb5fa5842b2aa3f6f4ac1869","Type":"ContainerStarted","Data":"7f1f47f9a0b2f92aa1e6c97604179ddca66ace8d0f9289c6b75fd3307a55099e"}
Sep 17 17:34:27.152901 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:27.152782    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-45dp8_bd527736-9920-4c79-9069-1eb8be07f869/ovn-acl-logging/0.log"
Sep 17 17:34:27.153289 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:27.153263    2574 generic.go:356] "Generic (PLEG): container finished" podID="bd527736-9920-4c79-9069-1eb8be07f869" containerID="3f6b2f280dedeac6eabb09b80e8b30f757d4feb9eda418a0bdf3a6f37c84e071" exitCode=1
Sep 17 17:34:27.153370 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:27.153315    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-45dp8" event={"ID":"bd527736-9920-4c79-9069-1eb8be07f869","Type":"ContainerStarted","Data":"63ffbc45825908a9ff25b72ecf9e1735d6010013280579110b268d9ece905775"}
Sep 17 17:34:27.153420 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:27.153385    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-45dp8" event={"ID":"bd527736-9920-4c79-9069-1eb8be07f869","Type":"ContainerStarted","Data":"cf4b757591522cd2ad5e00c6909f13654de598aeb44cd6258ac1a3e04cfd2d65"}
Sep 17 17:34:27.153464 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:27.153428    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-45dp8" event={"ID":"bd527736-9920-4c79-9069-1eb8be07f869","Type":"ContainerDied","Data":"3f6b2f280dedeac6eabb09b80e8b30f757d4feb9eda418a0bdf3a6f37c84e071"}
Sep 17 17:34:27.153513 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:27.153467    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-45dp8" event={"ID":"bd527736-9920-4c79-9069-1eb8be07f869","Type":"ContainerStarted","Data":"cdca4697d494a51a1a82fce589bb6af7ecc9146503729da799160b9b1f84ae1a"}
Sep 17 17:34:27.158845 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:27.158809    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-n28f2" event={"ID":"94a7bbd1-e769-4046-a56e-442208fb2ca2","Type":"ContainerStarted","Data":"bf53fa34d35f9bbfb8720d9e461560ca219de7662446ca42aedf4a1158af69fd"}
Sep 17 17:34:27.163157 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:27.163135    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-ck426" event={"ID":"f33e242b-1311-43e7-8a5f-a97ad057236f","Type":"ContainerStarted","Data":"a53d7d3253e8f3045096f3128ff2c7a856bd59ccf30afd0f537692d5365b4aa8"}
Sep 17 17:34:27.190104 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:27.189917    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-n28f2" podStartSLOduration=2.169590314 podStartE2EDuration="20.189896472s" totalImagesPullingTime="18.020306158s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:34:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:34:27.189414222 +0000 UTC m=+22.378287934" watchObservedRunningTime="2026-09-17 17:34:27.189896472 +0000 UTC m=+22.378770179"
Sep 17 17:34:27.190337 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:27.190298    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-96.ec2.internal" podStartSLOduration=20.190290942 podStartE2EDuration="20.190290942s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:34:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:34:27.165636169 +0000 UTC m=+22.354509877" watchObservedRunningTime="2026-09-17 17:34:27.190290942 +0000 UTC m=+22.379164649"
Sep 17 17:34:27.948224 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:27.948154    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j9d2w"
Sep 17 17:34:27.948365 ip-10-0-134-96 kubenswrapper[2574]: E0917 17:34:27.948258    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j9d2w" podUID="61c43e2e-f36c-401d-a10c-ff5f88fab086"
Sep 17 17:34:27.948365 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:27.948268    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4tpm5"
Sep 17 17:34:27.948448 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:27.948285    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sv2qq"
Sep 17 17:34:27.948448 ip-10-0-134-96 kubenswrapper[2574]: E0917 17:34:27.948398    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4tpm5" podUID="c18eb096-7089-426a-957b-3dc1a2d7e837"
Sep 17 17:34:27.948520 ip-10-0-134-96 kubenswrapper[2574]: E0917 17:34:27.948454    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sv2qq" podUID="fadd68fa-727c-4581-bed0-5ab8b73b9982"
Sep 17 17:34:28.167597 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:28.167574    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-45dp8_bd527736-9920-4c79-9069-1eb8be07f869/ovn-acl-logging/0.log"
Sep 17 17:34:28.168357 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:28.167908    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-45dp8" event={"ID":"bd527736-9920-4c79-9069-1eb8be07f869","Type":"ContainerStarted","Data":"b911d7d20c586f345a03fa4cd53209dd8675a9642ddd317afd71c2ddf891e09d"}
Sep 17 17:34:28.168357 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:28.167983    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-45dp8" event={"ID":"bd527736-9920-4c79-9069-1eb8be07f869","Type":"ContainerStarted","Data":"ba6dd4c46a50d24524592013582b2b1f8e751580f56190338c3de2816883589a"}
Sep 17 17:34:28.169441 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:28.169351    2574 generic.go:356] "Generic (PLEG): container finished" podID="d7f1aae7-8961-4863-abff-c2259ef65ede" containerID="82c74985ffa66b6865e8f2f63f88d5d2154181a351b67de29c069c03438c51cd" exitCode=0
Sep 17 17:34:28.169441 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:28.169402    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-g8nlc" event={"ID":"d7f1aae7-8961-4863-abff-c2259ef65ede","Type":"ContainerDied","Data":"82c74985ffa66b6865e8f2f63f88d5d2154181a351b67de29c069c03438c51cd"}
Sep 17 17:34:28.173601 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:28.173558    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-2j4ds" event={"ID":"d3f553e8-b416-4d7d-bbef-991a0404b287","Type":"ContainerStarted","Data":"2d10c661fe520c51378143e27bf70d7ae3dba88b0214da712395e854a8bf296b"}
Sep 17 17:34:28.175381 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:28.175357    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rp7jd" event={"ID":"ff8b66be-df38-4e3f-9080-483b295fa79c","Type":"ContainerStarted","Data":"b5378e435802b80d7637b5a661fa349e5b308487f8e3c0f6c6cd8093011f1e57"}
Sep 17 17:34:28.176761 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:28.176741    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-xkj8c" event={"ID":"12264c60-6721-4820-816f-798eefa04bd5","Type":"ContainerStarted","Data":"7b6c7242eaa1e2733a467e69102c5abca21a7c7dd3c72ab20380618333bd4c49"}
Sep 17 17:34:28.178614 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:28.178593    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-t79st" event={"ID":"67b5a7ce-5a80-4224-9f54-86ac40fa136b","Type":"ContainerStarted","Data":"d7bfede63d05b24a1f550441ad88ba286ceedafd3c453303423e868006437ff7"}
Sep 17 17:34:28.179821 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:28.179801    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-lf4tf" event={"ID":"31495fae-ecfa-4263-a3db-702f75935118","Type":"ContainerStarted","Data":"e59267c5190cc40dca089016daa7f616981c2357adda258c4158effa01ace360"}
Sep 17 17:34:28.181161 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:28.181138    2574 generic.go:356] "Generic (PLEG): container finished" podID="5ecb72cbea487238ef07d1e2c3698290" containerID="905f4aef6beb7f76cb1b14823136e7f56bcf13d863d06fcd8d1f47d83c0ddbc4" exitCode=0
Sep 17 17:34:28.181249 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:28.181184    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-96.ec2.internal" event={"ID":"5ecb72cbea487238ef07d1e2c3698290","Type":"ContainerDied","Data":"905f4aef6beb7f76cb1b14823136e7f56bcf13d863d06fcd8d1f47d83c0ddbc4"}
Sep 17 17:34:28.196526 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:28.196493    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-ck426" podStartSLOduration=2.4265485 podStartE2EDuration="21.196483006s" totalImagesPullingTime="18.769934506s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:34:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:34:27.211219881 +0000 UTC m=+22.400093590" watchObservedRunningTime="2026-09-17 17:34:28.196483006 +0000 UTC m=+23.385356713"
Sep 17 17:34:28.215799 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:28.215622    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-2j4ds" podStartSLOduration=2.777666141 podStartE2EDuration="21.215608797s" totalImagesPullingTime="18.437942656s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:34:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:34:28.215420316 +0000 UTC m=+23.404294023" watchObservedRunningTime="2026-09-17 17:34:28.215608797 +0000 UTC m=+23.404482530"
Sep 17 17:34:28.235365 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:28.235322    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-lf4tf" podStartSLOduration=2.83919578 podStartE2EDuration="21.235305225s" totalImagesPullingTime="18.396109445s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:34:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:34:28.23510567 +0000 UTC m=+23.423979379" watchObservedRunningTime="2026-09-17 17:34:28.235305225 +0000 UTC m=+23.424178932"
Sep 17 17:34:28.274124 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:28.274085    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-t79st" podStartSLOduration=3.776141966 podStartE2EDuration="22.274070299s" totalImagesPullingTime="18.497928333s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:34:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:34:28.273657304 +0000 UTC m=+23.462531011" watchObservedRunningTime="2026-09-17 17:34:28.274070299 +0000 UTC m=+23.462944005"
Sep 17 17:34:28.291037 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:28.290995    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-xkj8c" podStartSLOduration=2.809382963 podStartE2EDuration="21.290985745s" totalImagesPullingTime="18.481602782s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:34:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:34:28.290836052 +0000 UTC m=+23.479709759" watchObservedRunningTime="2026-09-17 17:34:28.290985745 +0000 UTC m=+23.479859496"
Sep 17 17:34:28.295163 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:28.295125    2574 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 17 17:34:28.451268 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:28.451236    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j4ds_d3f553e8-b416-4d7d-bbef-991a0404b287/konnectivity-agent/0.log" limit=1024
Sep 17 17:34:28.527668 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:28.527640    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j4ds_d3f553e8-b416-4d7d-bbef-991a0404b287/konnectivity-agent/0.log" limit=1024
Sep 17 17:34:28.603998 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:28.603968    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j4ds_d3f553e8-b416-4d7d-bbef-991a0404b287/konnectivity-agent/0.log" limit=1024
Sep 17 17:34:28.682653 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:28.682623    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j4ds_d3f553e8-b416-4d7d-bbef-991a0404b287/konnectivity-agent/0.log" limit=1024
Sep 17 17:34:28.763974 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:28.763944    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j4ds_d3f553e8-b416-4d7d-bbef-991a0404b287/konnectivity-agent/0.log" limit=1024
Sep 17 17:34:28.879090 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:28.878948    2574 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-17T17:34:28.295142823Z","UUID":"70b85df0-ce2d-4ec9-9928-5cac6b392ca8","Handler":null,"Name":"","Endpoint":""}
Sep 17 17:34:28.882699 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:28.882678    2574 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 17 17:34:28.882843 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:28.882709    2574 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 17 17:34:29.185229 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:29.185191    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rp7jd" event={"ID":"ff8b66be-df38-4e3f-9080-483b295fa79c","Type":"ContainerStarted","Data":"b27ce1480548953b59844b5e99fca3f5f7dc1d3de71272e81272164e1c27be88"}
Sep 17 17:34:29.187141 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:29.187095    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-96.ec2.internal" event={"ID":"5ecb72cbea487238ef07d1e2c3698290","Type":"ContainerStarted","Data":"c72f36aa530c350bb898e0576190b8fee0ec0ede4c598ed4a147872f074d946b"}
Sep 17 17:34:29.205364 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:29.205272    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-96.ec2.internal" podStartSLOduration=22.205253794 podStartE2EDuration="22.205253794s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:34:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:34:29.204889969 +0000 UTC m=+24.393763675" watchObservedRunningTime="2026-09-17 17:34:29.205253794 +0000 UTC m=+24.394127503"
Sep 17 17:34:29.948297 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:29.948255    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4tpm5"
Sep 17 17:34:29.948297 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:29.948296    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sv2qq"
Sep 17 17:34:29.948541 ip-10-0-134-96 kubenswrapper[2574]: E0917 17:34:29.948386    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4tpm5" podUID="c18eb096-7089-426a-957b-3dc1a2d7e837"
Sep 17 17:34:29.948541 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:29.948396    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j9d2w"
Sep 17 17:34:29.948541 ip-10-0-134-96 kubenswrapper[2574]: E0917 17:34:29.948490    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sv2qq" podUID="fadd68fa-727c-4581-bed0-5ab8b73b9982"
Sep 17 17:34:29.948677 ip-10-0-134-96 kubenswrapper[2574]: E0917 17:34:29.948583    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j9d2w" podUID="61c43e2e-f36c-401d-a10c-ff5f88fab086"
Sep 17 17:34:30.193650 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:30.193617    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-45dp8_bd527736-9920-4c79-9069-1eb8be07f869/ovn-acl-logging/0.log"
Sep 17 17:34:30.194166 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:30.194143    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-45dp8" event={"ID":"bd527736-9920-4c79-9069-1eb8be07f869","Type":"ContainerStarted","Data":"b95b9584c7fa5afbe70bc89bf857dcf57e3e3ba22276e0f174b5b5e2ea38a7e5"}
Sep 17 17:34:30.196546 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:30.196518    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rp7jd" event={"ID":"ff8b66be-df38-4e3f-9080-483b295fa79c","Type":"ContainerStarted","Data":"868e98dd87831e3c1a8623d637754aa6caf7957c4aeea8670e59b1bfd9c971a9"}
Sep 17 17:34:30.220981 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:30.220904    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rp7jd" podStartSLOduration=3.3762238079999998 podStartE2EDuration="23.220886582s" totalImagesPullingTime="19.844662774s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:34:07 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:34:30.220276723 +0000 UTC m=+25.409150429" watchObservedRunningTime="2026-09-17 17:34:30.220886582 +0000 UTC m=+25.409760290"
Sep 17 17:34:30.555063 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:30.554972    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-2j4ds"
Sep 17 17:34:30.555618 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:30.555594    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-2j4ds"
Sep 17 17:34:31.181364 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:31.181335    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j4ds_d3f553e8-b416-4d7d-bbef-991a0404b287/konnectivity-agent/0.log" limit=1024
Sep 17 17:34:31.199029 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:31.199000    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-2j4ds"
Sep 17 17:34:31.199542 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:31.199434    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-2j4ds"
Sep 17 17:34:31.276136 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:31.276101    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j4ds_d3f553e8-b416-4d7d-bbef-991a0404b287/konnectivity-agent/0.log" limit=1024
Sep 17 17:34:31.358724 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:31.358666    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j4ds_d3f553e8-b416-4d7d-bbef-991a0404b287/konnectivity-agent/0.log" limit=1024
Sep 17 17:34:31.948944 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:31.948907    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4tpm5"
Sep 17 17:34:31.948944 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:31.948933    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j9d2w"
Sep 17 17:34:31.949199 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:31.948933    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sv2qq"
Sep 17 17:34:31.949199 ip-10-0-134-96 kubenswrapper[2574]: E0917 17:34:31.949071    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4tpm5" podUID="c18eb096-7089-426a-957b-3dc1a2d7e837"
Sep 17 17:34:31.949288 ip-10-0-134-96 kubenswrapper[2574]: E0917 17:34:31.949205    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j9d2w" podUID="61c43e2e-f36c-401d-a10c-ff5f88fab086"
Sep 17 17:34:31.949342 ip-10-0-134-96 kubenswrapper[2574]: E0917 17:34:31.949305    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sv2qq" podUID="fadd68fa-727c-4581-bed0-5ab8b73b9982"
Sep 17 17:34:32.205078 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:32.204945    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-45dp8_bd527736-9920-4c79-9069-1eb8be07f869/ovn-acl-logging/0.log"
Sep 17 17:34:32.205525 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:32.205492    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-45dp8" event={"ID":"bd527736-9920-4c79-9069-1eb8be07f869","Type":"ContainerStarted","Data":"0a5445b8fb036b7530aba85f7a335b99ffbbbe0523d99f0fe9d7e33556d8c92e"}
Sep 17 17:34:32.205993 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:32.205975    2574 scope.go:117] "RemoveContainer" containerID="3f6b2f280dedeac6eabb09b80e8b30f757d4feb9eda418a0bdf3a6f37c84e071"
Sep 17 17:34:33.210866 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:33.210843    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-45dp8_bd527736-9920-4c79-9069-1eb8be07f869/ovn-acl-logging/0.log"
Sep 17 17:34:33.211521 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:33.211254    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-45dp8" event={"ID":"bd527736-9920-4c79-9069-1eb8be07f869","Type":"ContainerStarted","Data":"bac0604d06c853e44a7b67f36f48bf5fe9556cc860c64e31428d3f433823fdf1"}
Sep 17 17:34:33.211521 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:33.211442    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-45dp8"
Sep 17 17:34:33.211521 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:33.211514    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-45dp8"
Sep 17 17:34:33.211653 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:33.211567    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-45dp8"
Sep 17 17:34:33.213487 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:33.213459    2574 generic.go:356] "Generic (PLEG): container finished" podID="d7f1aae7-8961-4863-abff-c2259ef65ede" containerID="30a2d1a383f77cf3a45789e661f75ae41a590815afb3d5e5c9507ae68e62c08d" exitCode=0
Sep 17 17:34:33.213610 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:33.213538    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-g8nlc" event={"ID":"d7f1aae7-8961-4863-abff-c2259ef65ede","Type":"ContainerDied","Data":"30a2d1a383f77cf3a45789e661f75ae41a590815afb3d5e5c9507ae68e62c08d"}
Sep 17 17:34:33.226306 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:33.226284    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-45dp8"
Sep 17 17:34:33.226400 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:33.226375    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-45dp8"
Sep 17 17:34:33.243827 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:33.241856    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-45dp8" podStartSLOduration=7.832496219 podStartE2EDuration="26.241832375s" totalImagesPullingTime="18.409336156s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:34:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:34:33.240459966 +0000 UTC m=+28.429333672" watchObservedRunningTime="2026-09-17 17:34:33.241832375 +0000 UTC m=+28.430706080"
Sep 17 17:34:33.948714 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:33.948693    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4tpm5"
Sep 17 17:34:33.948827 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:33.948792    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sv2qq"
Sep 17 17:34:33.948827 ip-10-0-134-96 kubenswrapper[2574]: E0917 17:34:33.948796    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4tpm5" podUID="c18eb096-7089-426a-957b-3dc1a2d7e837"
Sep 17 17:34:33.948827 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:33.948822    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j9d2w"
Sep 17 17:34:33.948951 ip-10-0-134-96 kubenswrapper[2574]: E0917 17:34:33.948921    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sv2qq" podUID="fadd68fa-727c-4581-bed0-5ab8b73b9982"
Sep 17 17:34:33.949024 ip-10-0-134-96 kubenswrapper[2574]: E0917 17:34:33.949003    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j9d2w" podUID="61c43e2e-f36c-401d-a10c-ff5f88fab086"
Sep 17 17:34:34.217295 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:34.217204    2574 generic.go:356] "Generic (PLEG): container finished" podID="d7f1aae7-8961-4863-abff-c2259ef65ede" containerID="6d3483d4819cf9a04423c12d8de4ab1bccf3cd5ae245fae72a2c6c82b2b0ee34" exitCode=0
Sep 17 17:34:34.217749 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:34.217300    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-g8nlc" event={"ID":"d7f1aae7-8961-4863-abff-c2259ef65ede","Type":"ContainerDied","Data":"6d3483d4819cf9a04423c12d8de4ab1bccf3cd5ae245fae72a2c6c82b2b0ee34"}
Sep 17 17:34:34.346411 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:34.346384    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-sv2qq"]
Sep 17 17:34:34.346546 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:34.346479    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sv2qq"
Sep 17 17:34:34.346585 ip-10-0-134-96 kubenswrapper[2574]: E0917 17:34:34.346564    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sv2qq" podUID="fadd68fa-727c-4581-bed0-5ab8b73b9982"
Sep 17 17:34:34.349747 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:34.349718    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-j9d2w"]
Sep 17 17:34:34.349885 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:34.349805    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j9d2w"
Sep 17 17:34:34.349957 ip-10-0-134-96 kubenswrapper[2574]: E0917 17:34:34.349897    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j9d2w" podUID="61c43e2e-f36c-401d-a10c-ff5f88fab086"
Sep 17 17:34:34.350316 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:34.350294    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-4tpm5"]
Sep 17 17:34:34.350421 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:34.350366    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4tpm5"
Sep 17 17:34:34.350496 ip-10-0-134-96 kubenswrapper[2574]: E0917 17:34:34.350438    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4tpm5" podUID="c18eb096-7089-426a-957b-3dc1a2d7e837"
Sep 17 17:34:35.221681 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:35.221649    2574 generic.go:356] "Generic (PLEG): container finished" podID="d7f1aae7-8961-4863-abff-c2259ef65ede" containerID="a628910afb8c0e83a23788d4cab92e1f5f5da243a6e839503cafcb7d04c731ba" exitCode=0
Sep 17 17:34:35.222032 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:35.221740    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-g8nlc" event={"ID":"d7f1aae7-8961-4863-abff-c2259ef65ede","Type":"ContainerDied","Data":"a628910afb8c0e83a23788d4cab92e1f5f5da243a6e839503cafcb7d04c731ba"}
Sep 17 17:34:35.948763 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:35.948683    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4tpm5"
Sep 17 17:34:35.948763 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:35.948728    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sv2qq"
Sep 17 17:34:35.948975 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:35.948789    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j9d2w"
Sep 17 17:34:35.948975 ip-10-0-134-96 kubenswrapper[2574]: E0917 17:34:35.948806    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4tpm5" podUID="c18eb096-7089-426a-957b-3dc1a2d7e837"
Sep 17 17:34:35.948975 ip-10-0-134-96 kubenswrapper[2574]: E0917 17:34:35.948881    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j9d2w" podUID="61c43e2e-f36c-401d-a10c-ff5f88fab086"
Sep 17 17:34:35.948975 ip-10-0-134-96 kubenswrapper[2574]: E0917 17:34:35.948968    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sv2qq" podUID="fadd68fa-727c-4581-bed0-5ab8b73b9982"
Sep 17 17:34:37.377085 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:37.377034    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j4ds_d3f553e8-b416-4d7d-bbef-991a0404b287/konnectivity-agent/0.log" limit=1024
Sep 17 17:34:37.474915 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:37.474883    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j4ds_d3f553e8-b416-4d7d-bbef-991a0404b287/konnectivity-agent/0.log" limit=1024
Sep 17 17:34:37.948059 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:37.948019    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4tpm5"
Sep 17 17:34:37.948255 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:37.948064    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j9d2w"
Sep 17 17:34:37.948255 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:37.948064    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sv2qq"
Sep 17 17:34:37.948255 ip-10-0-134-96 kubenswrapper[2574]: E0917 17:34:37.948171    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4tpm5" podUID="c18eb096-7089-426a-957b-3dc1a2d7e837"
Sep 17 17:34:37.948255 ip-10-0-134-96 kubenswrapper[2574]: E0917 17:34:37.948224    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j9d2w" podUID="61c43e2e-f36c-401d-a10c-ff5f88fab086"
Sep 17 17:34:37.948463 ip-10-0-134-96 kubenswrapper[2574]: E0917 17:34:37.948307    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sv2qq" podUID="fadd68fa-727c-4581-bed0-5ab8b73b9982"
Sep 17 17:34:39.579457 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:39.579227    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-96.ec2.internal" event="NodeReady"
Sep 17 17:34:39.579779 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:39.579605    2574 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 17 17:34:39.633925 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:39.633895    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fadd68fa-727c-4581-bed0-5ab8b73b9982-original-pull-secret\") pod \"global-pull-secret-syncer-sv2qq\" (UID: \"fadd68fa-727c-4581-bed0-5ab8b73b9982\") " pod="kube-system/global-pull-secret-syncer-sv2qq"
Sep 17 17:34:39.634117 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:39.633944    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c18eb096-7089-426a-957b-3dc1a2d7e837-metrics-certs\") pod \"network-metrics-daemon-4tpm5\" (UID: \"c18eb096-7089-426a-957b-3dc1a2d7e837\") " pod="openshift-multus/network-metrics-daemon-4tpm5"
Sep 17 17:34:39.634117 ip-10-0-134-96 kubenswrapper[2574]: E0917 17:34:39.634096    2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:34:39.634243 ip-10-0-134-96 kubenswrapper[2574]: E0917 17:34:39.634157    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c18eb096-7089-426a-957b-3dc1a2d7e837-metrics-certs podName:c18eb096-7089-426a-957b-3dc1a2d7e837 nodeName:}" failed. No retries permitted until 2026-09-17 17:35:11.634138619 +0000 UTC m=+66.823012306 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c18eb096-7089-426a-957b-3dc1a2d7e837-metrics-certs") pod "network-metrics-daemon-4tpm5" (UID: "c18eb096-7089-426a-957b-3dc1a2d7e837") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:34:39.634243 ip-10-0-134-96 kubenswrapper[2574]: E0917 17:34:39.634096    2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 17:34:39.634350 ip-10-0-134-96 kubenswrapper[2574]: E0917 17:34:39.634277    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fadd68fa-727c-4581-bed0-5ab8b73b9982-original-pull-secret podName:fadd68fa-727c-4581-bed0-5ab8b73b9982 nodeName:}" failed. No retries permitted until 2026-09-17 17:35:11.634258665 +0000 UTC m=+66.823132366 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fadd68fa-727c-4581-bed0-5ab8b73b9982-original-pull-secret") pod "global-pull-secret-syncer-sv2qq" (UID: "fadd68fa-727c-4581-bed0-5ab8b73b9982") : object "kube-system"/"original-pull-secret" not registered
Sep 17 17:34:39.634667 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:39.634649    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-tnkgs"]
Sep 17 17:34:39.651750 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:39.651729    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-2sz2c"]
Sep 17 17:34:39.651897 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:39.651880    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-tnkgs"
Sep 17 17:34:39.654285 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:39.654248    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 17 17:34:39.654554 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:39.654250    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-bd5nq\""
Sep 17 17:34:39.654700 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:39.654678    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 17 17:34:39.672293 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:39.672263    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-tnkgs"]
Sep 17 17:34:39.672401 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:39.672308    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-2sz2c"]
Sep 17 17:34:39.672401 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:39.672377    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-2sz2c"
Sep 17 17:34:39.674661 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:39.674639    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 17 17:34:39.674774 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:39.674671    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 17 17:34:39.674774 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:39.674693    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-hrfbx\""
Sep 17 17:34:39.674774 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:39.674693    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 17 17:34:39.734793 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:39.734765    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fz57p\" (UniqueName: \"kubernetes.io/projected/0f867410-d783-47d8-91aa-564f7e5cecc8-kube-api-access-fz57p\") pod \"ingress-canary-2sz2c\" (UID: \"0f867410-d783-47d8-91aa-564f7e5cecc8\") " pod="openshift-ingress-canary/ingress-canary-2sz2c"
Sep 17 17:34:39.734900 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:39.734799    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/d4f6642f-b0fd-4434-a318-530eac3af0a0-config-volume\") pod \"dns-default-tnkgs\" (UID: \"d4f6642f-b0fd-4434-a318-530eac3af0a0\") " pod="openshift-dns/dns-default-tnkgs"
Sep 17 17:34:39.734900 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:39.734823    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0f867410-d783-47d8-91aa-564f7e5cecc8-cert\") pod \"ingress-canary-2sz2c\" (UID: \"0f867410-d783-47d8-91aa-564f7e5cecc8\") " pod="openshift-ingress-canary/ingress-canary-2sz2c"
Sep 17 17:34:39.734900 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:39.734837    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f8fr4\" (UniqueName: \"kubernetes.io/projected/d4f6642f-b0fd-4434-a318-530eac3af0a0-kube-api-access-f8fr4\") pod \"dns-default-tnkgs\" (UID: \"d4f6642f-b0fd-4434-a318-530eac3af0a0\") " pod="openshift-dns/dns-default-tnkgs"
Sep 17 17:34:39.734900 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:39.734875    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2lk56\" (UniqueName: \"kubernetes.io/projected/61c43e2e-f36c-401d-a10c-ff5f88fab086-kube-api-access-2lk56\") pod \"network-check-target-j9d2w\" (UID: \"61c43e2e-f36c-401d-a10c-ff5f88fab086\") " pod="openshift-network-diagnostics/network-check-target-j9d2w"
Sep 17 17:34:39.735037 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:39.734904    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4f6642f-b0fd-4434-a318-530eac3af0a0-tmp-dir\") pod \"dns-default-tnkgs\" (UID: \"d4f6642f-b0fd-4434-a318-530eac3af0a0\") " pod="openshift-dns/dns-default-tnkgs"
Sep 17 17:34:39.735037 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:39.734937    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d4f6642f-b0fd-4434-a318-530eac3af0a0-metrics-tls\") pod \"dns-default-tnkgs\" (UID: \"d4f6642f-b0fd-4434-a318-530eac3af0a0\") " pod="openshift-dns/dns-default-tnkgs"
Sep 17 17:34:39.735037 ip-10-0-134-96 kubenswrapper[2574]: E0917 17:34:39.734992    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 17:34:39.735037 ip-10-0-134-96 kubenswrapper[2574]: E0917 17:34:39.735012    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 17:34:39.735037 ip-10-0-134-96 kubenswrapper[2574]: E0917 17:34:39.735020    2574 projected.go:196] Error preparing data for projected volume kube-api-access-2lk56 for pod openshift-network-diagnostics/network-check-target-j9d2w: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:34:39.735224 ip-10-0-134-96 kubenswrapper[2574]: E0917 17:34:39.735084    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/61c43e2e-f36c-401d-a10c-ff5f88fab086-kube-api-access-2lk56 podName:61c43e2e-f36c-401d-a10c-ff5f88fab086 nodeName:}" failed. No retries permitted until 2026-09-17 17:35:11.735068507 +0000 UTC m=+66.923942192 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-2lk56" (UniqueName: "kubernetes.io/projected/61c43e2e-f36c-401d-a10c-ff5f88fab086-kube-api-access-2lk56") pod "network-check-target-j9d2w" (UID: "61c43e2e-f36c-401d-a10c-ff5f88fab086") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:34:39.802917 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:39.802886    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j4ds_d3f553e8-b416-4d7d-bbef-991a0404b287/konnectivity-agent/0.log" limit=1024
Sep 17 17:34:39.835922 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:39.835900    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4f6642f-b0fd-4434-a318-530eac3af0a0-tmp-dir\") pod \"dns-default-tnkgs\" (UID: \"d4f6642f-b0fd-4434-a318-530eac3af0a0\") " pod="openshift-dns/dns-default-tnkgs"
Sep 17 17:34:39.836018 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:39.835946    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d4f6642f-b0fd-4434-a318-530eac3af0a0-metrics-tls\") pod \"dns-default-tnkgs\" (UID: \"d4f6642f-b0fd-4434-a318-530eac3af0a0\") " pod="openshift-dns/dns-default-tnkgs"
Sep 17 17:34:39.836018 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:39.835978    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fz57p\" (UniqueName: \"kubernetes.io/projected/0f867410-d783-47d8-91aa-564f7e5cecc8-kube-api-access-fz57p\") pod \"ingress-canary-2sz2c\" (UID: \"0f867410-d783-47d8-91aa-564f7e5cecc8\") " pod="openshift-ingress-canary/ingress-canary-2sz2c"
Sep 17 17:34:39.836018 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:39.836011    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/d4f6642f-b0fd-4434-a318-530eac3af0a0-config-volume\") pod \"dns-default-tnkgs\" (UID: \"d4f6642f-b0fd-4434-a318-530eac3af0a0\") " pod="openshift-dns/dns-default-tnkgs"
Sep 17 17:34:39.836201 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:39.836062    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0f867410-d783-47d8-91aa-564f7e5cecc8-cert\") pod \"ingress-canary-2sz2c\" (UID: \"0f867410-d783-47d8-91aa-564f7e5cecc8\") " pod="openshift-ingress-canary/ingress-canary-2sz2c"
Sep 17 17:34:39.836201 ip-10-0-134-96 kubenswrapper[2574]: E0917 17:34:39.836093    2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 17:34:39.836201 ip-10-0-134-96 kubenswrapper[2574]: E0917 17:34:39.836159    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d4f6642f-b0fd-4434-a318-530eac3af0a0-metrics-tls podName:d4f6642f-b0fd-4434-a318-530eac3af0a0 nodeName:}" failed. No retries permitted until 2026-09-17 17:34:40.336139917 +0000 UTC m=+35.525013619 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d4f6642f-b0fd-4434-a318-530eac3af0a0-metrics-tls") pod "dns-default-tnkgs" (UID: "d4f6642f-b0fd-4434-a318-530eac3af0a0") : secret "dns-default-metrics-tls" not found
Sep 17 17:34:39.836336 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:39.836281    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f8fr4\" (UniqueName: \"kubernetes.io/projected/d4f6642f-b0fd-4434-a318-530eac3af0a0-kube-api-access-f8fr4\") pod \"dns-default-tnkgs\" (UID: \"d4f6642f-b0fd-4434-a318-530eac3af0a0\") " pod="openshift-dns/dns-default-tnkgs"
Sep 17 17:34:39.836552 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:39.836537    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4f6642f-b0fd-4434-a318-530eac3af0a0-tmp-dir\") pod \"dns-default-tnkgs\" (UID: \"d4f6642f-b0fd-4434-a318-530eac3af0a0\") " pod="openshift-dns/dns-default-tnkgs"
Sep 17 17:34:39.836794 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:39.836777    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/d4f6642f-b0fd-4434-a318-530eac3af0a0-config-volume\") pod \"dns-default-tnkgs\" (UID: \"d4f6642f-b0fd-4434-a318-530eac3af0a0\") " pod="openshift-dns/dns-default-tnkgs"
Sep 17 17:34:39.840052 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:39.840019    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0f867410-d783-47d8-91aa-564f7e5cecc8-cert\") pod \"ingress-canary-2sz2c\" (UID: \"0f867410-d783-47d8-91aa-564f7e5cecc8\") " pod="openshift-ingress-canary/ingress-canary-2sz2c"
Sep 17 17:34:39.847587 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:39.847567    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f8fr4\" (UniqueName: \"kubernetes.io/projected/d4f6642f-b0fd-4434-a318-530eac3af0a0-kube-api-access-f8fr4\") pod \"dns-default-tnkgs\" (UID: \"d4f6642f-b0fd-4434-a318-530eac3af0a0\") " pod="openshift-dns/dns-default-tnkgs"
Sep 17 17:34:39.847739 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:39.847721    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fz57p\" (UniqueName: \"kubernetes.io/projected/0f867410-d783-47d8-91aa-564f7e5cecc8-kube-api-access-fz57p\") pod \"ingress-canary-2sz2c\" (UID: \"0f867410-d783-47d8-91aa-564f7e5cecc8\") " pod="openshift-ingress-canary/ingress-canary-2sz2c"
Sep 17 17:34:39.901145 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:39.901109    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j4ds_d3f553e8-b416-4d7d-bbef-991a0404b287/konnectivity-agent/0.log" limit=1024
Sep 17 17:34:39.948895 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:39.948840    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4tpm5"
Sep 17 17:34:39.948978 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:39.948935    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j9d2w"
Sep 17 17:34:39.949038 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:39.948845    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sv2qq"
Sep 17 17:34:39.951514 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:39.951498    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-qcgkp\""
Sep 17 17:34:39.951599 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:39.951569    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 17 17:34:39.951827 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:39.951813    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 17 17:34:39.951903 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:39.951848    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-crvff\""
Sep 17 17:34:39.951903 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:39.951893    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 17 17:34:39.952007 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:39.951894    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 17 17:34:40.019007 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:40.018988    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-2sz2c"
Sep 17 17:34:40.236022 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:40.235940    2574 generic.go:356] "Generic (PLEG): container finished" podID="d7f1aae7-8961-4863-abff-c2259ef65ede" containerID="ad26d587ca0b356d16f18db7638cf5fa928a1c539d7e202a7cfa6b69cbb1d3c1" exitCode=0
Sep 17 17:34:40.236022 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:40.235993    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-g8nlc" event={"ID":"d7f1aae7-8961-4863-abff-c2259ef65ede","Type":"ContainerDied","Data":"ad26d587ca0b356d16f18db7638cf5fa928a1c539d7e202a7cfa6b69cbb1d3c1"}
Sep 17 17:34:40.339615 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:40.339516    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-2sz2c"]
Sep 17 17:34:40.339862 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:40.339843    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d4f6642f-b0fd-4434-a318-530eac3af0a0-metrics-tls\") pod \"dns-default-tnkgs\" (UID: \"d4f6642f-b0fd-4434-a318-530eac3af0a0\") " pod="openshift-dns/dns-default-tnkgs"
Sep 17 17:34:40.340217 ip-10-0-134-96 kubenswrapper[2574]: E0917 17:34:40.339998    2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 17:34:40.340217 ip-10-0-134-96 kubenswrapper[2574]: E0917 17:34:40.340107    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d4f6642f-b0fd-4434-a318-530eac3af0a0-metrics-tls podName:d4f6642f-b0fd-4434-a318-530eac3af0a0 nodeName:}" failed. No retries permitted until 2026-09-17 17:34:41.340084309 +0000 UTC m=+36.528958011 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d4f6642f-b0fd-4434-a318-530eac3af0a0-metrics-tls") pod "dns-default-tnkgs" (UID: "d4f6642f-b0fd-4434-a318-530eac3af0a0") : secret "dns-default-metrics-tls" not found
Sep 17 17:34:40.346911 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:40.346891    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f867410_d783_47d8_91aa_564f7e5cecc8.slice/crio-94ae1d07dde617aff24b2fe0691fbd768271ee98649fb035fd9072444daa5e35 WatchSource:0}: Error finding container 94ae1d07dde617aff24b2fe0691fbd768271ee98649fb035fd9072444daa5e35: Status 404 returned error can't find the container with id 94ae1d07dde617aff24b2fe0691fbd768271ee98649fb035fd9072444daa5e35
Sep 17 17:34:40.526669 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:40.526644    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j4ds_d3f553e8-b416-4d7d-bbef-991a0404b287/konnectivity-agent/0.log" limit=1024
Sep 17 17:34:40.626581 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:40.626554    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j4ds_d3f553e8-b416-4d7d-bbef-991a0404b287/konnectivity-agent/0.log" limit=1024
Sep 17 17:34:40.705736 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:40.705707    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j4ds_d3f553e8-b416-4d7d-bbef-991a0404b287/konnectivity-agent/0.log" limit=1024
Sep 17 17:34:41.239790 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:41.239756    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-2sz2c" event={"ID":"0f867410-d783-47d8-91aa-564f7e5cecc8","Type":"ContainerStarted","Data":"94ae1d07dde617aff24b2fe0691fbd768271ee98649fb035fd9072444daa5e35"}
Sep 17 17:34:41.243225 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:41.243196    2574 generic.go:356] "Generic (PLEG): container finished" podID="d7f1aae7-8961-4863-abff-c2259ef65ede" containerID="827b5060dcbbf35a9811c3d19444f6343861ecbc0bd5256951d9dd31f4224423" exitCode=0
Sep 17 17:34:41.243361 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:41.243258    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-g8nlc" event={"ID":"d7f1aae7-8961-4863-abff-c2259ef65ede","Type":"ContainerDied","Data":"827b5060dcbbf35a9811c3d19444f6343861ecbc0bd5256951d9dd31f4224423"}
Sep 17 17:34:41.349647 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:41.349171    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d4f6642f-b0fd-4434-a318-530eac3af0a0-metrics-tls\") pod \"dns-default-tnkgs\" (UID: \"d4f6642f-b0fd-4434-a318-530eac3af0a0\") " pod="openshift-dns/dns-default-tnkgs"
Sep 17 17:34:41.349647 ip-10-0-134-96 kubenswrapper[2574]: E0917 17:34:41.349354    2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 17:34:41.349647 ip-10-0-134-96 kubenswrapper[2574]: E0917 17:34:41.349476    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d4f6642f-b0fd-4434-a318-530eac3af0a0-metrics-tls podName:d4f6642f-b0fd-4434-a318-530eac3af0a0 nodeName:}" failed. No retries permitted until 2026-09-17 17:34:43.34945537 +0000 UTC m=+38.538329060 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d4f6642f-b0fd-4434-a318-530eac3af0a0-metrics-tls") pod "dns-default-tnkgs" (UID: "d4f6642f-b0fd-4434-a318-530eac3af0a0") : secret "dns-default-metrics-tls" not found
Sep 17 17:34:42.250202 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:42.250164    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-g8nlc" event={"ID":"d7f1aae7-8961-4863-abff-c2259ef65ede","Type":"ContainerStarted","Data":"80f342221aba1c114c15e94c6bb794627aba369b7800575e6ce55f5d1d0b26ae"}
Sep 17 17:34:42.278265 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:42.278198    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-g8nlc" podStartSLOduration=7.892289698 podStartE2EDuration="35.27817514s" totalImagesPullingTime="27.385885442s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:34:07 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:34:42.277816505 +0000 UTC m=+37.466690213" watchObservedRunningTime="2026-09-17 17:34:42.27817514 +0000 UTC m=+37.467048847"
Sep 17 17:34:42.402606 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:42.402573    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j4ds_d3f553e8-b416-4d7d-bbef-991a0404b287/konnectivity-agent/0.log" limit=1024
Sep 17 17:34:42.471832 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:42.471794    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j4ds_d3f553e8-b416-4d7d-bbef-991a0404b287/konnectivity-agent/0.log" limit=1024
Sep 17 17:34:43.253902 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:43.253873    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-2sz2c" event={"ID":"0f867410-d783-47d8-91aa-564f7e5cecc8","Type":"ContainerStarted","Data":"2d6504f5a2c757e836a7299fd50ea304b9acb45a7253d42922e29b5afb608c7a"}
Sep 17 17:34:43.276556 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:43.276510    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-2sz2c" podStartSLOduration=1.805532687 podStartE2EDuration="4.276489171s" totalImagesPullingTime="2.470956484s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:34:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:34:43.276122146 +0000 UTC m=+38.464995852" watchObservedRunningTime="2026-09-17 17:34:43.276489171 +0000 UTC m=+38.465362875"
Sep 17 17:34:43.363583 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:43.363547    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d4f6642f-b0fd-4434-a318-530eac3af0a0-metrics-tls\") pod \"dns-default-tnkgs\" (UID: \"d4f6642f-b0fd-4434-a318-530eac3af0a0\") " pod="openshift-dns/dns-default-tnkgs"
Sep 17 17:34:43.363809 ip-10-0-134-96 kubenswrapper[2574]: E0917 17:34:43.363788    2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 17:34:43.363893 ip-10-0-134-96 kubenswrapper[2574]: E0917 17:34:43.363880    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d4f6642f-b0fd-4434-a318-530eac3af0a0-metrics-tls podName:d4f6642f-b0fd-4434-a318-530eac3af0a0 nodeName:}" failed. No retries permitted until 2026-09-17 17:34:47.363858765 +0000 UTC m=+42.552732467 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d4f6642f-b0fd-4434-a318-530eac3af0a0-metrics-tls") pod "dns-default-tnkgs" (UID: "d4f6642f-b0fd-4434-a318-530eac3af0a0") : secret "dns-default-metrics-tls" not found
Sep 17 17:34:45.225083 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:45.225035    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j4ds_d3f553e8-b416-4d7d-bbef-991a0404b287/konnectivity-agent/0.log" limit=1024
Sep 17 17:34:45.306415 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:45.306386    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j4ds_d3f553e8-b416-4d7d-bbef-991a0404b287/konnectivity-agent/0.log" limit=1024
Sep 17 17:34:45.669186 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:45.669154    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-ns2j6"]
Sep 17 17:34:45.679420 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:45.679401    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-ns2j6"
Sep 17 17:34:45.682467 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:45.682441    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Sep 17 17:34:45.682630 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:45.682465    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Sep 17 17:34:45.682773 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:45.682509    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Sep 17 17:34:45.682864 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:45.682551    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Sep 17 17:34:45.683018 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:45.683003    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-ns2j6"]
Sep 17 17:34:45.683237 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:45.683224    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-nbfq9\""
Sep 17 17:34:45.774604 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:45.774580    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-5ct2x"]
Sep 17 17:34:45.779027 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:45.779008    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/88497892-29d0-4b92-803b-2ce9ebebf402-serving-cert\") pod \"kube-storage-version-migrator-operator-55889f7b47-ns2j6\" (UID: \"88497892-29d0-4b92-803b-2ce9ebebf402\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-ns2j6"
Sep 17 17:34:45.779145 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:45.779053    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/88497892-29d0-4b92-803b-2ce9ebebf402-config\") pod \"kube-storage-version-migrator-operator-55889f7b47-ns2j6\" (UID: \"88497892-29d0-4b92-803b-2ce9ebebf402\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-ns2j6"
Sep 17 17:34:45.779145 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:45.779074    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mp28b\" (UniqueName: \"kubernetes.io/projected/88497892-29d0-4b92-803b-2ce9ebebf402-kube-api-access-mp28b\") pod \"kube-storage-version-migrator-operator-55889f7b47-ns2j6\" (UID: \"88497892-29d0-4b92-803b-2ce9ebebf402\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-ns2j6"
Sep 17 17:34:45.786582 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:45.786568    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-5ct2x"
Sep 17 17:34:45.789086 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:45.789071    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 17 17:34:45.790089 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:45.790067    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Sep 17 17:34:45.790308 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:45.790294    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 17 17:34:45.790382 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:45.790307    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-zbmkl\""
Sep 17 17:34:45.790382 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:45.790347    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Sep 17 17:34:45.794684 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:45.794665    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-5ct2x"]
Sep 17 17:34:45.879785 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:45.879756    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-6f9c644dd9-d8vvv"]
Sep 17 17:34:45.879942 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:45.879924    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/88497892-29d0-4b92-803b-2ce9ebebf402-serving-cert\") pod \"kube-storage-version-migrator-operator-55889f7b47-ns2j6\" (UID: \"88497892-29d0-4b92-803b-2ce9ebebf402\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-ns2j6"
Sep 17 17:34:45.880007 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:45.879954    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/88497892-29d0-4b92-803b-2ce9ebebf402-config\") pod \"kube-storage-version-migrator-operator-55889f7b47-ns2j6\" (UID: \"88497892-29d0-4b92-803b-2ce9ebebf402\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-ns2j6"
Sep 17 17:34:45.880007 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:45.879987    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/b3d9c617-b694-40c2-98ce-3772a892df3c-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-5ct2x\" (UID: \"b3d9c617-b694-40c2-98ce-3772a892df3c\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-5ct2x"
Sep 17 17:34:45.880130 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:45.880100    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mp28b\" (UniqueName: \"kubernetes.io/projected/88497892-29d0-4b92-803b-2ce9ebebf402-kube-api-access-mp28b\") pod \"kube-storage-version-migrator-operator-55889f7b47-ns2j6\" (UID: \"88497892-29d0-4b92-803b-2ce9ebebf402\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-ns2j6"
Sep 17 17:34:45.880185 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:45.880129    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rgwqd\" (UniqueName: \"kubernetes.io/projected/b3d9c617-b694-40c2-98ce-3772a892df3c-kube-api-access-rgwqd\") pod \"cluster-monitoring-operator-847b8bd8d7-5ct2x\" (UID: \"b3d9c617-b694-40c2-98ce-3772a892df3c\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-5ct2x"
Sep 17 17:34:45.880239 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:45.880214    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/b3d9c617-b694-40c2-98ce-3772a892df3c-telemetry-config\") pod \"cluster-monitoring-operator-847b8bd8d7-5ct2x\" (UID: \"b3d9c617-b694-40c2-98ce-3772a892df3c\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-5ct2x"
Sep 17 17:34:45.880551 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:45.880529    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/88497892-29d0-4b92-803b-2ce9ebebf402-config\") pod \"kube-storage-version-migrator-operator-55889f7b47-ns2j6\" (UID: \"88497892-29d0-4b92-803b-2ce9ebebf402\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-ns2j6"
Sep 17 17:34:45.883325 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:45.883305    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/88497892-29d0-4b92-803b-2ce9ebebf402-serving-cert\") pod \"kube-storage-version-migrator-operator-55889f7b47-ns2j6\" (UID: \"88497892-29d0-4b92-803b-2ce9ebebf402\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-ns2j6"
Sep 17 17:34:45.889223 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:45.889206    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-6f9c644dd9-d8vvv"
Sep 17 17:34:45.892341 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:45.892325    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 17 17:34:45.892518 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:45.892493    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 17 17:34:45.892518 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:45.892510    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Sep 17 17:34:45.892647 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:45.892565    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-2kxf4\""
Sep 17 17:34:45.892816 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:45.892802    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Sep 17 17:34:45.894498 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:45.894478    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mp28b\" (UniqueName: \"kubernetes.io/projected/88497892-29d0-4b92-803b-2ce9ebebf402-kube-api-access-mp28b\") pod \"kube-storage-version-migrator-operator-55889f7b47-ns2j6\" (UID: \"88497892-29d0-4b92-803b-2ce9ebebf402\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-ns2j6"
Sep 17 17:34:45.901001 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:45.898154    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Sep 17 17:34:45.903211 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:45.903190    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-6f9c644dd9-d8vvv"]
Sep 17 17:34:45.968757 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:45.968700    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-7cbc45fddc-9jr2n"]
Sep 17 17:34:45.980640 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:45.980612    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/b3d9c617-b694-40c2-98ce-3772a892df3c-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-5ct2x\" (UID: \"b3d9c617-b694-40c2-98ce-3772a892df3c\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-5ct2x"
Sep 17 17:34:45.980754 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:45.980657    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rgwqd\" (UniqueName: \"kubernetes.io/projected/b3d9c617-b694-40c2-98ce-3772a892df3c-kube-api-access-rgwqd\") pod \"cluster-monitoring-operator-847b8bd8d7-5ct2x\" (UID: \"b3d9c617-b694-40c2-98ce-3772a892df3c\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-5ct2x"
Sep 17 17:34:45.980812 ip-10-0-134-96 kubenswrapper[2574]: E0917 17:34:45.980779    2574 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 17 17:34:45.980849 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:45.980834    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d0d18822-7421-4563-8c4d-777f26ae6c0e-serving-cert\") pod \"insights-operator-6f9c644dd9-d8vvv\" (UID: \"d0d18822-7421-4563-8c4d-777f26ae6c0e\") " pod="openshift-insights/insights-operator-6f9c644dd9-d8vvv"
Sep 17 17:34:45.980887 ip-10-0-134-96 kubenswrapper[2574]: E0917 17:34:45.980868    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b3d9c617-b694-40c2-98ce-3772a892df3c-cluster-monitoring-operator-tls podName:b3d9c617-b694-40c2-98ce-3772a892df3c nodeName:}" failed. No retries permitted until 2026-09-17 17:34:46.480849833 +0000 UTC m=+41.669723532 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/b3d9c617-b694-40c2-98ce-3772a892df3c-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-5ct2x" (UID: "b3d9c617-b694-40c2-98ce-3772a892df3c") : secret "cluster-monitoring-operator-tls" not found
Sep 17 17:34:45.980936 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:45.980918    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d0d18822-7421-4563-8c4d-777f26ae6c0e-service-ca-bundle\") pod \"insights-operator-6f9c644dd9-d8vvv\" (UID: \"d0d18822-7421-4563-8c4d-777f26ae6c0e\") " pod="openshift-insights/insights-operator-6f9c644dd9-d8vvv"
Sep 17 17:34:45.980978 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:45.980943    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k8ff9\" (UniqueName: \"kubernetes.io/projected/d0d18822-7421-4563-8c4d-777f26ae6c0e-kube-api-access-k8ff9\") pod \"insights-operator-6f9c644dd9-d8vvv\" (UID: \"d0d18822-7421-4563-8c4d-777f26ae6c0e\") " pod="openshift-insights/insights-operator-6f9c644dd9-d8vvv"
Sep 17 17:34:45.981021 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:45.980987    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/b3d9c617-b694-40c2-98ce-3772a892df3c-telemetry-config\") pod \"cluster-monitoring-operator-847b8bd8d7-5ct2x\" (UID: \"b3d9c617-b694-40c2-98ce-3772a892df3c\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-5ct2x"
Sep 17 17:34:45.981087 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:45.981029    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d0d18822-7421-4563-8c4d-777f26ae6c0e-tmp\") pod \"insights-operator-6f9c644dd9-d8vvv\" (UID: \"d0d18822-7421-4563-8c4d-777f26ae6c0e\") " pod="openshift-insights/insights-operator-6f9c644dd9-d8vvv"
Sep 17 17:34:45.981087 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:45.981064    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/d0d18822-7421-4563-8c4d-777f26ae6c0e-snapshots\") pod \"insights-operator-6f9c644dd9-d8vvv\" (UID: \"d0d18822-7421-4563-8c4d-777f26ae6c0e\") " pod="openshift-insights/insights-operator-6f9c644dd9-d8vvv"
Sep 17 17:34:45.981167 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:45.981125    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d0d18822-7421-4563-8c4d-777f26ae6c0e-trusted-ca-bundle\") pod \"insights-operator-6f9c644dd9-d8vvv\" (UID: \"d0d18822-7421-4563-8c4d-777f26ae6c0e\") " pod="openshift-insights/insights-operator-6f9c644dd9-d8vvv"
Sep 17 17:34:45.981597 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:45.981576    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/b3d9c617-b694-40c2-98ce-3772a892df3c-telemetry-config\") pod \"cluster-monitoring-operator-847b8bd8d7-5ct2x\" (UID: \"b3d9c617-b694-40c2-98ce-3772a892df3c\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-5ct2x"
Sep 17 17:34:45.981936 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:45.981919    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-57cdc4b768-gvwc6"]
Sep 17 17:34:45.982066 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:45.982038    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-9jr2n"
Sep 17 17:34:45.984626 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:45.984610    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-7r58n\""
Sep 17 17:34:45.989481 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:45.989459    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-ns2j6"
Sep 17 17:34:45.994850 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:45.994833    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7cbc45fddc-9jr2n"]
Sep 17 17:34:45.994936 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:45.994861    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-57cdc4b768-gvwc6"]
Sep 17 17:34:45.994936 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:45.994919    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-gvwc6"
Sep 17 17:34:45.996555 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:45.996539    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rgwqd\" (UniqueName: \"kubernetes.io/projected/b3d9c617-b694-40c2-98ce-3772a892df3c-kube-api-access-rgwqd\") pod \"cluster-monitoring-operator-847b8bd8d7-5ct2x\" (UID: \"b3d9c617-b694-40c2-98ce-3772a892df3c\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-5ct2x"
Sep 17 17:34:45.997480 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:45.997350    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Sep 17 17:34:45.997480 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:45.997380    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Sep 17 17:34:45.997480 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:45.997379    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Sep 17 17:34:45.997480 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:45.997402    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Sep 17 17:34:45.997701 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:45.997492    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-sh6rg\""
Sep 17 17:34:46.015845 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:46.015827    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j4ds_d3f553e8-b416-4d7d-bbef-991a0404b287/konnectivity-agent/0.log" limit=1024
Sep 17 17:34:46.081682 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:46.081657    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d0d18822-7421-4563-8c4d-777f26ae6c0e-service-ca-bundle\") pod \"insights-operator-6f9c644dd9-d8vvv\" (UID: \"d0d18822-7421-4563-8c4d-777f26ae6c0e\") " pod="openshift-insights/insights-operator-6f9c644dd9-d8vvv"
Sep 17 17:34:46.081801 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:46.081687    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8ff9\" (UniqueName: \"kubernetes.io/projected/d0d18822-7421-4563-8c4d-777f26ae6c0e-kube-api-access-k8ff9\") pod \"insights-operator-6f9c644dd9-d8vvv\" (UID: \"d0d18822-7421-4563-8c4d-777f26ae6c0e\") " pod="openshift-insights/insights-operator-6f9c644dd9-d8vvv"
Sep 17 17:34:46.081801 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:46.081760    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b7aea158-3b08-4631-93f5-c03c3dfb7998-config\") pod \"service-ca-operator-57cdc4b768-gvwc6\" (UID: \"b7aea158-3b08-4631-93f5-c03c3dfb7998\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-gvwc6"
Sep 17 17:34:46.081873 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:46.081808    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d0d18822-7421-4563-8c4d-777f26ae6c0e-tmp\") pod \"insights-operator-6f9c644dd9-d8vvv\" (UID: \"d0d18822-7421-4563-8c4d-777f26ae6c0e\") " pod="openshift-insights/insights-operator-6f9c644dd9-d8vvv"
Sep 17 17:34:46.082027 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:46.082009    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/d0d18822-7421-4563-8c4d-777f26ae6c0e-snapshots\") pod \"insights-operator-6f9c644dd9-d8vvv\" (UID: \"d0d18822-7421-4563-8c4d-777f26ae6c0e\") " pod="openshift-insights/insights-operator-6f9c644dd9-d8vvv"
Sep 17 17:34:46.082134 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:46.082057    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rxrx6\" (UniqueName: \"kubernetes.io/projected/b7aea158-3b08-4631-93f5-c03c3dfb7998-kube-api-access-rxrx6\") pod \"service-ca-operator-57cdc4b768-gvwc6\" (UID: \"b7aea158-3b08-4631-93f5-c03c3dfb7998\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-gvwc6"
Sep 17 17:34:46.082134 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:46.082099    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d0d18822-7421-4563-8c4d-777f26ae6c0e-trusted-ca-bundle\") pod \"insights-operator-6f9c644dd9-d8vvv\" (UID: \"d0d18822-7421-4563-8c4d-777f26ae6c0e\") " pod="openshift-insights/insights-operator-6f9c644dd9-d8vvv"
Sep 17 17:34:46.082217 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:46.082195    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d0d18822-7421-4563-8c4d-777f26ae6c0e-serving-cert\") pod \"insights-operator-6f9c644dd9-d8vvv\" (UID: \"d0d18822-7421-4563-8c4d-777f26ae6c0e\") " pod="openshift-insights/insights-operator-6f9c644dd9-d8vvv"
Sep 17 17:34:46.082258 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:46.082236    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fjcg7\" (UniqueName: \"kubernetes.io/projected/49831c33-9661-4705-a0e9-a24da9ea059a-kube-api-access-fjcg7\") pod \"network-check-source-7cbc45fddc-9jr2n\" (UID: \"49831c33-9661-4705-a0e9-a24da9ea059a\") " pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-9jr2n"
Sep 17 17:34:46.082297 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:46.082198    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d0d18822-7421-4563-8c4d-777f26ae6c0e-tmp\") pod \"insights-operator-6f9c644dd9-d8vvv\" (UID: \"d0d18822-7421-4563-8c4d-777f26ae6c0e\") " pod="openshift-insights/insights-operator-6f9c644dd9-d8vvv"
Sep 17 17:34:46.082329 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:46.082265    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b7aea158-3b08-4631-93f5-c03c3dfb7998-serving-cert\") pod \"service-ca-operator-57cdc4b768-gvwc6\" (UID: \"b7aea158-3b08-4631-93f5-c03c3dfb7998\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-gvwc6"
Sep 17 17:34:46.082368 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:46.082332    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d0d18822-7421-4563-8c4d-777f26ae6c0e-service-ca-bundle\") pod \"insights-operator-6f9c644dd9-d8vvv\" (UID: \"d0d18822-7421-4563-8c4d-777f26ae6c0e\") " pod="openshift-insights/insights-operator-6f9c644dd9-d8vvv"
Sep 17 17:34:46.082518 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:46.082497    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/d0d18822-7421-4563-8c4d-777f26ae6c0e-snapshots\") pod \"insights-operator-6f9c644dd9-d8vvv\" (UID: \"d0d18822-7421-4563-8c4d-777f26ae6c0e\") " pod="openshift-insights/insights-operator-6f9c644dd9-d8vvv"
Sep 17 17:34:46.082789 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:46.082770    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d0d18822-7421-4563-8c4d-777f26ae6c0e-trusted-ca-bundle\") pod \"insights-operator-6f9c644dd9-d8vvv\" (UID: \"d0d18822-7421-4563-8c4d-777f26ae6c0e\") " pod="openshift-insights/insights-operator-6f9c644dd9-d8vvv"
Sep 17 17:34:46.084258 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:46.084234    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d0d18822-7421-4563-8c4d-777f26ae6c0e-serving-cert\") pod \"insights-operator-6f9c644dd9-d8vvv\" (UID: \"d0d18822-7421-4563-8c4d-777f26ae6c0e\") " pod="openshift-insights/insights-operator-6f9c644dd9-d8vvv"
Sep 17 17:34:46.087400 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:46.087381    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j4ds_d3f553e8-b416-4d7d-bbef-991a0404b287/konnectivity-agent/0.log" limit=1024
Sep 17 17:34:46.091913 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:46.091892    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k8ff9\" (UniqueName: \"kubernetes.io/projected/d0d18822-7421-4563-8c4d-777f26ae6c0e-kube-api-access-k8ff9\") pod \"insights-operator-6f9c644dd9-d8vvv\" (UID: \"d0d18822-7421-4563-8c4d-777f26ae6c0e\") " pod="openshift-insights/insights-operator-6f9c644dd9-d8vvv"
Sep 17 17:34:46.183161 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:46.183135    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fjcg7\" (UniqueName: \"kubernetes.io/projected/49831c33-9661-4705-a0e9-a24da9ea059a-kube-api-access-fjcg7\") pod \"network-check-source-7cbc45fddc-9jr2n\" (UID: \"49831c33-9661-4705-a0e9-a24da9ea059a\") " pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-9jr2n"
Sep 17 17:34:46.183161 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:46.183164    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b7aea158-3b08-4631-93f5-c03c3dfb7998-serving-cert\") pod \"service-ca-operator-57cdc4b768-gvwc6\" (UID: \"b7aea158-3b08-4631-93f5-c03c3dfb7998\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-gvwc6"
Sep 17 17:34:46.183394 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:46.183375    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b7aea158-3b08-4631-93f5-c03c3dfb7998-config\") pod \"service-ca-operator-57cdc4b768-gvwc6\" (UID: \"b7aea158-3b08-4631-93f5-c03c3dfb7998\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-gvwc6"
Sep 17 17:34:46.183465 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:46.183421    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rxrx6\" (UniqueName: \"kubernetes.io/projected/b7aea158-3b08-4631-93f5-c03c3dfb7998-kube-api-access-rxrx6\") pod \"service-ca-operator-57cdc4b768-gvwc6\" (UID: \"b7aea158-3b08-4631-93f5-c03c3dfb7998\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-gvwc6"
Sep 17 17:34:46.183925 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:46.183908    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b7aea158-3b08-4631-93f5-c03c3dfb7998-config\") pod \"service-ca-operator-57cdc4b768-gvwc6\" (UID: \"b7aea158-3b08-4631-93f5-c03c3dfb7998\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-gvwc6"
Sep 17 17:34:46.185082 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:46.185068    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b7aea158-3b08-4631-93f5-c03c3dfb7998-serving-cert\") pod \"service-ca-operator-57cdc4b768-gvwc6\" (UID: \"b7aea158-3b08-4631-93f5-c03c3dfb7998\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-gvwc6"
Sep 17 17:34:46.192789 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:46.192771    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rxrx6\" (UniqueName: \"kubernetes.io/projected/b7aea158-3b08-4631-93f5-c03c3dfb7998-kube-api-access-rxrx6\") pod \"service-ca-operator-57cdc4b768-gvwc6\" (UID: \"b7aea158-3b08-4631-93f5-c03c3dfb7998\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-gvwc6"
Sep 17 17:34:46.193123 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:46.193107    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fjcg7\" (UniqueName: \"kubernetes.io/projected/49831c33-9661-4705-a0e9-a24da9ea059a-kube-api-access-fjcg7\") pod \"network-check-source-7cbc45fddc-9jr2n\" (UID: \"49831c33-9661-4705-a0e9-a24da9ea059a\") " pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-9jr2n"
Sep 17 17:34:46.204900 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:46.204881    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-6f9c644dd9-d8vvv"
Sep 17 17:34:46.273252 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:46.273228    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-ns2j6"]
Sep 17 17:34:46.276269 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:46.276236    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88497892_29d0_4b92_803b_2ce9ebebf402.slice/crio-e032ab0caa84547541a7601d0066ddec854ffef85b4004bb83ca93d0085f5abf WatchSource:0}: Error finding container e032ab0caa84547541a7601d0066ddec854ffef85b4004bb83ca93d0085f5abf: Status 404 returned error can't find the container with id e032ab0caa84547541a7601d0066ddec854ffef85b4004bb83ca93d0085f5abf
Sep 17 17:34:46.290837 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:46.290820    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-9jr2n"
Sep 17 17:34:46.306316 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:46.306294    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-gvwc6"
Sep 17 17:34:46.485178 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:46.485154    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-6f9c644dd9-d8vvv"]
Sep 17 17:34:46.485454 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:46.485440    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/b3d9c617-b694-40c2-98ce-3772a892df3c-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-5ct2x\" (UID: \"b3d9c617-b694-40c2-98ce-3772a892df3c\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-5ct2x"
Sep 17 17:34:46.485569 ip-10-0-134-96 kubenswrapper[2574]: E0917 17:34:46.485545    2574 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 17 17:34:46.485601 ip-10-0-134-96 kubenswrapper[2574]: E0917 17:34:46.485595    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b3d9c617-b694-40c2-98ce-3772a892df3c-cluster-monitoring-operator-tls podName:b3d9c617-b694-40c2-98ce-3772a892df3c nodeName:}" failed. No retries permitted until 2026-09-17 17:34:47.485582839 +0000 UTC m=+42.674456525 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/b3d9c617-b694-40c2-98ce-3772a892df3c-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-5ct2x" (UID: "b3d9c617-b694-40c2-98ce-3772a892df3c") : secret "cluster-monitoring-operator-tls" not found
Sep 17 17:34:46.487507 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:46.487482    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0d18822_7421_4563_8c4d_777f26ae6c0e.slice/crio-80932627556a1230e6c7c9d43c016346b55ffbb8525b44a9a8f861cb5e51530d WatchSource:0}: Error finding container 80932627556a1230e6c7c9d43c016346b55ffbb8525b44a9a8f861cb5e51530d: Status 404 returned error can't find the container with id 80932627556a1230e6c7c9d43c016346b55ffbb8525b44a9a8f861cb5e51530d
Sep 17 17:34:46.566018 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:46.565995    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7cbc45fddc-9jr2n"]
Sep 17 17:34:46.568633 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:46.568614    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49831c33_9661_4705_a0e9_a24da9ea059a.slice/crio-72bfd25fcf44a81f7e95a9e8a68548d761092524aaeef74a314b67d7c1652861 WatchSource:0}: Error finding container 72bfd25fcf44a81f7e95a9e8a68548d761092524aaeef74a314b67d7c1652861: Status 404 returned error can't find the container with id 72bfd25fcf44a81f7e95a9e8a68548d761092524aaeef74a314b67d7c1652861
Sep 17 17:34:46.596780 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:46.596760    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-57cdc4b768-gvwc6"]
Sep 17 17:34:46.599478 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:46.599456    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7aea158_3b08_4631_93f5_c03c3dfb7998.slice/crio-22a993a727e8e79223f7717c354815b13ec8d2e8bebaad97e04c48e82fba3069 WatchSource:0}: Error finding container 22a993a727e8e79223f7717c354815b13ec8d2e8bebaad97e04c48e82fba3069: Status 404 returned error can't find the container with id 22a993a727e8e79223f7717c354815b13ec8d2e8bebaad97e04c48e82fba3069
Sep 17 17:34:47.268326 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:47.268266    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-gvwc6" event={"ID":"b7aea158-3b08-4631-93f5-c03c3dfb7998","Type":"ContainerStarted","Data":"22a993a727e8e79223f7717c354815b13ec8d2e8bebaad97e04c48e82fba3069"}
Sep 17 17:34:47.269831 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:47.269784    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-6f9c644dd9-d8vvv" event={"ID":"d0d18822-7421-4563-8c4d-777f26ae6c0e","Type":"ContainerStarted","Data":"80932627556a1230e6c7c9d43c016346b55ffbb8525b44a9a8f861cb5e51530d"}
Sep 17 17:34:47.271784 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:47.271761    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-9jr2n" event={"ID":"49831c33-9661-4705-a0e9-a24da9ea059a","Type":"ContainerStarted","Data":"72bfd25fcf44a81f7e95a9e8a68548d761092524aaeef74a314b67d7c1652861"}
Sep 17 17:34:47.276206 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:47.276086    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-ns2j6" event={"ID":"88497892-29d0-4b92-803b-2ce9ebebf402","Type":"ContainerStarted","Data":"e032ab0caa84547541a7601d0066ddec854ffef85b4004bb83ca93d0085f5abf"}
Sep 17 17:34:47.393900 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:47.393837    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d4f6642f-b0fd-4434-a318-530eac3af0a0-metrics-tls\") pod \"dns-default-tnkgs\" (UID: \"d4f6642f-b0fd-4434-a318-530eac3af0a0\") " pod="openshift-dns/dns-default-tnkgs"
Sep 17 17:34:47.394098 ip-10-0-134-96 kubenswrapper[2574]: E0917 17:34:47.394000    2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 17:34:47.394098 ip-10-0-134-96 kubenswrapper[2574]: E0917 17:34:47.394087    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d4f6642f-b0fd-4434-a318-530eac3af0a0-metrics-tls podName:d4f6642f-b0fd-4434-a318-530eac3af0a0 nodeName:}" failed. No retries permitted until 2026-09-17 17:34:55.394066831 +0000 UTC m=+50.582940517 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d4f6642f-b0fd-4434-a318-530eac3af0a0-metrics-tls") pod "dns-default-tnkgs" (UID: "d4f6642f-b0fd-4434-a318-530eac3af0a0") : secret "dns-default-metrics-tls" not found
Sep 17 17:34:47.495709 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:47.495256    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/b3d9c617-b694-40c2-98ce-3772a892df3c-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-5ct2x\" (UID: \"b3d9c617-b694-40c2-98ce-3772a892df3c\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-5ct2x"
Sep 17 17:34:47.495709 ip-10-0-134-96 kubenswrapper[2574]: E0917 17:34:47.495383    2574 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 17 17:34:47.495709 ip-10-0-134-96 kubenswrapper[2574]: E0917 17:34:47.495431    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b3d9c617-b694-40c2-98ce-3772a892df3c-cluster-monitoring-operator-tls podName:b3d9c617-b694-40c2-98ce-3772a892df3c nodeName:}" failed. No retries permitted until 2026-09-17 17:34:49.495417495 +0000 UTC m=+44.684291180 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/b3d9c617-b694-40c2-98ce-3772a892df3c-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-5ct2x" (UID: "b3d9c617-b694-40c2-98ce-3772a892df3c") : secret "cluster-monitoring-operator-tls" not found
Sep 17 17:34:47.532523 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:47.532457    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j4ds_d3f553e8-b416-4d7d-bbef-991a0404b287/konnectivity-agent/0.log" limit=1024
Sep 17 17:34:47.635318 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:47.635286    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j4ds_d3f553e8-b416-4d7d-bbef-991a0404b287/konnectivity-agent/0.log" limit=1024
Sep 17 17:34:49.513946 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:49.513905    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/b3d9c617-b694-40c2-98ce-3772a892df3c-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-5ct2x\" (UID: \"b3d9c617-b694-40c2-98ce-3772a892df3c\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-5ct2x"
Sep 17 17:34:49.514490 ip-10-0-134-96 kubenswrapper[2574]: E0917 17:34:49.514106    2574 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 17 17:34:49.514490 ip-10-0-134-96 kubenswrapper[2574]: E0917 17:34:49.514189    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b3d9c617-b694-40c2-98ce-3772a892df3c-cluster-monitoring-operator-tls podName:b3d9c617-b694-40c2-98ce-3772a892df3c nodeName:}" failed. No retries permitted until 2026-09-17 17:34:53.51416577 +0000 UTC m=+48.703039458 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/b3d9c617-b694-40c2-98ce-3772a892df3c-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-5ct2x" (UID: "b3d9c617-b694-40c2-98ce-3772a892df3c") : secret "cluster-monitoring-operator-tls" not found
Sep 17 17:34:51.289833 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:51.289739    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-ns2j6" event={"ID":"88497892-29d0-4b92-803b-2ce9ebebf402","Type":"ContainerStarted","Data":"2210b41930ca6c286cd8ee9c2380485d3323d99ab9ad7d12a294dfc5a896214d"}
Sep 17 17:34:51.291538 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:51.291508    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-gvwc6" event={"ID":"b7aea158-3b08-4631-93f5-c03c3dfb7998","Type":"ContainerStarted","Data":"12c35d710b9ed8098b09894dd6b614a82ce7cb3539631e9985c3e298f5aafc48"}
Sep 17 17:34:51.293170 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:51.293138    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-6f9c644dd9-d8vvv" event={"ID":"d0d18822-7421-4563-8c4d-777f26ae6c0e","Type":"ContainerStarted","Data":"06c8cc6f64e7d77b30ad6545da8eb495db3f3105558f0b3657b7908e7165fe51"}
Sep 17 17:34:51.310550 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:51.310499    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-ns2j6" podStartSLOduration=1.562772393 podStartE2EDuration="6.310481661s" totalImagesPullingTime="4.747709268s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:34:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:34:51.309981384 +0000 UTC m=+46.498855102" watchObservedRunningTime="2026-09-17 17:34:51.310481661 +0000 UTC m=+46.499355368"
Sep 17 17:34:51.333167 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:51.333108    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-6f9c644dd9-d8vvv" podStartSLOduration=1.789657753 podStartE2EDuration="6.333092223s" totalImagesPullingTime="4.54343447s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:34:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:34:51.331929512 +0000 UTC m=+46.520803220" watchObservedRunningTime="2026-09-17 17:34:51.333092223 +0000 UTC m=+46.521965928"
Sep 17 17:34:51.350083 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:51.350024    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-gvwc6" podStartSLOduration=1.9238746839999998 podStartE2EDuration="6.350012925s" totalImagesPullingTime="4.426138241s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:34:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:34:51.348895778 +0000 UTC m=+46.537769486" watchObservedRunningTime="2026-09-17 17:34:51.350012925 +0000 UTC m=+46.538886632"
Sep 17 17:34:52.154983 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:52.154948    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j4ds_d3f553e8-b416-4d7d-bbef-991a0404b287/konnectivity-agent/0.log" limit=1024
Sep 17 17:34:52.190696 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:52.190656    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-76794cbd6c-7d5q6"]
Sep 17 17:34:52.208825 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:52.208791    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-76794cbd6c-7d5q6"]
Sep 17 17:34:52.209007 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:52.208950    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-7d5q6"
Sep 17 17:34:52.212225 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:52.212101    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Sep 17 17:34:52.212225 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:52.212171    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Sep 17 17:34:52.212413 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:52.212377    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-pdmg4\""
Sep 17 17:34:52.238752 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:52.238725    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j4ds_d3f553e8-b416-4d7d-bbef-991a0404b287/konnectivity-agent/0.log" limit=1024
Sep 17 17:34:52.298674 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:52.298629    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-9jr2n" event={"ID":"49831c33-9661-4705-a0e9-a24da9ea059a","Type":"ContainerStarted","Data":"0ec55f887378f361214b32d310190a09dc0ccf91801fa8d50b721010c1034952"}
Sep 17 17:34:52.320216 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:52.320169    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-9jr2n" podStartSLOduration=2.502143957 podStartE2EDuration="7.320152486s" totalImagesPullingTime="4.818008529s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:34:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:34:52.319298329 +0000 UTC m=+47.508172041" watchObservedRunningTime="2026-09-17 17:34:52.320152486 +0000 UTC m=+47.509026193"
Sep 17 17:34:52.338736 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:52.338713    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xfgzg\" (UniqueName: \"kubernetes.io/projected/38d5d165-e37b-4a2f-9c2a-67cbb3267248-kube-api-access-xfgzg\") pod \"migrator-76794cbd6c-7d5q6\" (UID: \"38d5d165-e37b-4a2f-9c2a-67cbb3267248\") " pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-7d5q6"
Sep 17 17:34:52.388606 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:52.388559    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j4ds_d3f553e8-b416-4d7d-bbef-991a0404b287/konnectivity-agent/0.log" limit=1024
Sep 17 17:34:52.439970 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:52.439885    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xfgzg\" (UniqueName: \"kubernetes.io/projected/38d5d165-e37b-4a2f-9c2a-67cbb3267248-kube-api-access-xfgzg\") pod \"migrator-76794cbd6c-7d5q6\" (UID: \"38d5d165-e37b-4a2f-9c2a-67cbb3267248\") " pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-7d5q6"
Sep 17 17:34:52.450952 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:52.450922    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xfgzg\" (UniqueName: \"kubernetes.io/projected/38d5d165-e37b-4a2f-9c2a-67cbb3267248-kube-api-access-xfgzg\") pod \"migrator-76794cbd6c-7d5q6\" (UID: \"38d5d165-e37b-4a2f-9c2a-67cbb3267248\") " pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-7d5q6"
Sep 17 17:34:52.520945 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:52.520917    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-7d5q6"
Sep 17 17:34:52.538825 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:52.538800    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j4ds_d3f553e8-b416-4d7d-bbef-991a0404b287/konnectivity-agent/0.log" limit=1024
Sep 17 17:34:52.653037 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:52.653006    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j4ds_d3f553e8-b416-4d7d-bbef-991a0404b287/konnectivity-agent/0.log" limit=1024
Sep 17 17:34:52.816518 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:52.816485    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-kpnbq"]
Sep 17 17:34:52.823535 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:52.823505    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j4ds_d3f553e8-b416-4d7d-bbef-991a0404b287/konnectivity-agent/0.log" limit=1024
Sep 17 17:34:52.826549 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:52.826524    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38d5d165_e37b_4a2f_9c2a_67cbb3267248.slice/crio-740f1103cd5b13aa54706cecf331bb05eda080d6c89d9af4af929937094589cd WatchSource:0}: Error finding container 740f1103cd5b13aa54706cecf331bb05eda080d6c89d9af4af929937094589cd: Status 404 returned error can't find the container with id 740f1103cd5b13aa54706cecf331bb05eda080d6c89d9af4af929937094589cd
Sep 17 17:34:52.833302 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:52.833255    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-76794cbd6c-7d5q6"]
Sep 17 17:34:52.833400 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:52.833304    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-kpnbq"]
Sep 17 17:34:52.833459 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:52.833409    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-kpnbq"
Sep 17 17:34:52.835959 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:52.835935    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 17 17:34:52.835959 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:52.835947    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-7fflr\""
Sep 17 17:34:52.836361 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:52.836343    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 17 17:34:52.944640 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:52.944606    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/76b84a3e-8b66-432d-b9f3-4aeaf96d262f-crio-socket\") pod \"insights-runtime-extractor-kpnbq\" (UID: \"76b84a3e-8b66-432d-b9f3-4aeaf96d262f\") " pod="openshift-insights/insights-runtime-extractor-kpnbq"
Sep 17 17:34:52.944640 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:52.944644    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/76b84a3e-8b66-432d-b9f3-4aeaf96d262f-data-volume\") pod \"insights-runtime-extractor-kpnbq\" (UID: \"76b84a3e-8b66-432d-b9f3-4aeaf96d262f\") " pod="openshift-insights/insights-runtime-extractor-kpnbq"
Sep 17 17:34:52.944817 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:52.944666    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/76b84a3e-8b66-432d-b9f3-4aeaf96d262f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kpnbq\" (UID: \"76b84a3e-8b66-432d-b9f3-4aeaf96d262f\") " pod="openshift-insights/insights-runtime-extractor-kpnbq"
Sep 17 17:34:52.944817 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:52.944726    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-txqzs\" (UniqueName: \"kubernetes.io/projected/76b84a3e-8b66-432d-b9f3-4aeaf96d262f-kube-api-access-txqzs\") pod \"insights-runtime-extractor-kpnbq\" (UID: \"76b84a3e-8b66-432d-b9f3-4aeaf96d262f\") " pod="openshift-insights/insights-runtime-extractor-kpnbq"
Sep 17 17:34:52.944817 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:52.944793    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/76b84a3e-8b66-432d-b9f3-4aeaf96d262f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kpnbq\" (UID: \"76b84a3e-8b66-432d-b9f3-4aeaf96d262f\") " pod="openshift-insights/insights-runtime-extractor-kpnbq"
Sep 17 17:34:52.953962 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:52.953938    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j4ds_d3f553e8-b416-4d7d-bbef-991a0404b287/konnectivity-agent/0.log" limit=1024
Sep 17 17:34:53.045388 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:53.045356    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/76b84a3e-8b66-432d-b9f3-4aeaf96d262f-crio-socket\") pod \"insights-runtime-extractor-kpnbq\" (UID: \"76b84a3e-8b66-432d-b9f3-4aeaf96d262f\") " pod="openshift-insights/insights-runtime-extractor-kpnbq"
Sep 17 17:34:53.045536 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:53.045395    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/76b84a3e-8b66-432d-b9f3-4aeaf96d262f-data-volume\") pod \"insights-runtime-extractor-kpnbq\" (UID: \"76b84a3e-8b66-432d-b9f3-4aeaf96d262f\") " pod="openshift-insights/insights-runtime-extractor-kpnbq"
Sep 17 17:34:53.045536 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:53.045415    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/76b84a3e-8b66-432d-b9f3-4aeaf96d262f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kpnbq\" (UID: \"76b84a3e-8b66-432d-b9f3-4aeaf96d262f\") " pod="openshift-insights/insights-runtime-extractor-kpnbq"
Sep 17 17:34:53.045536 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:53.045509    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-txqzs\" (UniqueName: \"kubernetes.io/projected/76b84a3e-8b66-432d-b9f3-4aeaf96d262f-kube-api-access-txqzs\") pod \"insights-runtime-extractor-kpnbq\" (UID: \"76b84a3e-8b66-432d-b9f3-4aeaf96d262f\") " pod="openshift-insights/insights-runtime-extractor-kpnbq"
Sep 17 17:34:53.045716 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:53.045553    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/76b84a3e-8b66-432d-b9f3-4aeaf96d262f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kpnbq\" (UID: \"76b84a3e-8b66-432d-b9f3-4aeaf96d262f\") " pod="openshift-insights/insights-runtime-extractor-kpnbq"
Sep 17 17:34:53.045716 ip-10-0-134-96 kubenswrapper[2574]: E0917 17:34:53.045514    2574 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 17:34:53.045716 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:53.045597    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/76b84a3e-8b66-432d-b9f3-4aeaf96d262f-crio-socket\") pod \"insights-runtime-extractor-kpnbq\" (UID: \"76b84a3e-8b66-432d-b9f3-4aeaf96d262f\") " pod="openshift-insights/insights-runtime-extractor-kpnbq"
Sep 17 17:34:53.045716 ip-10-0-134-96 kubenswrapper[2574]: E0917 17:34:53.045637    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/76b84a3e-8b66-432d-b9f3-4aeaf96d262f-insights-runtime-extractor-tls podName:76b84a3e-8b66-432d-b9f3-4aeaf96d262f nodeName:}" failed. No retries permitted until 2026-09-17 17:34:53.545616711 +0000 UTC m=+48.734490413 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/76b84a3e-8b66-432d-b9f3-4aeaf96d262f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-kpnbq" (UID: "76b84a3e-8b66-432d-b9f3-4aeaf96d262f") : secret "insights-runtime-extractor-tls" not found
Sep 17 17:34:53.045716 ip-10-0-134-96 kubenswrapper[2574]: E0917 17:34:53.045658    2574 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 17:34:53.045716 ip-10-0-134-96 kubenswrapper[2574]: E0917 17:34:53.045720    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/76b84a3e-8b66-432d-b9f3-4aeaf96d262f-kube-rbac-proxy-cm podName:76b84a3e-8b66-432d-b9f3-4aeaf96d262f nodeName:}" failed. No retries permitted until 2026-09-17 17:34:53.545709129 +0000 UTC m=+48.734582814 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/76b84a3e-8b66-432d-b9f3-4aeaf96d262f-kube-rbac-proxy-cm") pod "insights-runtime-extractor-kpnbq" (UID: "76b84a3e-8b66-432d-b9f3-4aeaf96d262f") : configmap "kube-rbac-proxy" not found
Sep 17 17:34:53.045949 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:53.045768    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/76b84a3e-8b66-432d-b9f3-4aeaf96d262f-data-volume\") pod \"insights-runtime-extractor-kpnbq\" (UID: \"76b84a3e-8b66-432d-b9f3-4aeaf96d262f\") " pod="openshift-insights/insights-runtime-extractor-kpnbq"
Sep 17 17:34:53.053447 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:53.053415    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j4ds_d3f553e8-b416-4d7d-bbef-991a0404b287/konnectivity-agent/0.log" limit=1024
Sep 17 17:34:53.054298 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:53.054280    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-txqzs\" (UniqueName: \"kubernetes.io/projected/76b84a3e-8b66-432d-b9f3-4aeaf96d262f-kube-api-access-txqzs\") pod \"insights-runtime-extractor-kpnbq\" (UID: \"76b84a3e-8b66-432d-b9f3-4aeaf96d262f\") " pod="openshift-insights/insights-runtime-extractor-kpnbq"
Sep 17 17:34:53.225788 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:53.225712    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j4ds_d3f553e8-b416-4d7d-bbef-991a0404b287/konnectivity-agent/0.log" limit=1024
Sep 17 17:34:53.302314 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:53.302283    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-7d5q6" event={"ID":"38d5d165-e37b-4a2f-9c2a-67cbb3267248","Type":"ContainerStarted","Data":"740f1103cd5b13aa54706cecf331bb05eda080d6c89d9af4af929937094589cd"}
Sep 17 17:34:53.321172 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:53.321148    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j4ds_d3f553e8-b416-4d7d-bbef-991a0404b287/konnectivity-agent/0.log" limit=1024
Sep 17 17:34:53.550300 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:53.550262    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/76b84a3e-8b66-432d-b9f3-4aeaf96d262f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kpnbq\" (UID: \"76b84a3e-8b66-432d-b9f3-4aeaf96d262f\") " pod="openshift-insights/insights-runtime-extractor-kpnbq"
Sep 17 17:34:53.550536 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:53.550319    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/b3d9c617-b694-40c2-98ce-3772a892df3c-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-5ct2x\" (UID: \"b3d9c617-b694-40c2-98ce-3772a892df3c\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-5ct2x"
Sep 17 17:34:53.550536 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:53.550358    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/76b84a3e-8b66-432d-b9f3-4aeaf96d262f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kpnbq\" (UID: \"76b84a3e-8b66-432d-b9f3-4aeaf96d262f\") " pod="openshift-insights/insights-runtime-extractor-kpnbq"
Sep 17 17:34:53.550536 ip-10-0-134-96 kubenswrapper[2574]: E0917 17:34:53.550437    2574 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 17:34:53.550536 ip-10-0-134-96 kubenswrapper[2574]: E0917 17:34:53.550458    2574 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 17 17:34:53.550536 ip-10-0-134-96 kubenswrapper[2574]: E0917 17:34:53.550510    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/76b84a3e-8b66-432d-b9f3-4aeaf96d262f-insights-runtime-extractor-tls podName:76b84a3e-8b66-432d-b9f3-4aeaf96d262f nodeName:}" failed. No retries permitted until 2026-09-17 17:34:54.550490046 +0000 UTC m=+49.739363754 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/76b84a3e-8b66-432d-b9f3-4aeaf96d262f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-kpnbq" (UID: "76b84a3e-8b66-432d-b9f3-4aeaf96d262f") : secret "insights-runtime-extractor-tls" not found
Sep 17 17:34:53.550536 ip-10-0-134-96 kubenswrapper[2574]: E0917 17:34:53.550531    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b3d9c617-b694-40c2-98ce-3772a892df3c-cluster-monitoring-operator-tls podName:b3d9c617-b694-40c2-98ce-3772a892df3c nodeName:}" failed. No retries permitted until 2026-09-17 17:35:01.550519902 +0000 UTC m=+56.739393588 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/b3d9c617-b694-40c2-98ce-3772a892df3c-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-5ct2x" (UID: "b3d9c617-b694-40c2-98ce-3772a892df3c") : secret "cluster-monitoring-operator-tls" not found
Sep 17 17:34:53.550813 ip-10-0-134-96 kubenswrapper[2574]: E0917 17:34:53.550560    2574 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 17:34:53.550813 ip-10-0-134-96 kubenswrapper[2574]: E0917 17:34:53.550634    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/76b84a3e-8b66-432d-b9f3-4aeaf96d262f-kube-rbac-proxy-cm podName:76b84a3e-8b66-432d-b9f3-4aeaf96d262f nodeName:}" failed. No retries permitted until 2026-09-17 17:34:54.550617377 +0000 UTC m=+49.739491076 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/76b84a3e-8b66-432d-b9f3-4aeaf96d262f-kube-rbac-proxy-cm") pod "insights-runtime-extractor-kpnbq" (UID: "76b84a3e-8b66-432d-b9f3-4aeaf96d262f") : configmap "kube-rbac-proxy" not found
Sep 17 17:34:54.416026 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:54.415993    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5fdf588c86-cxcfh"]
Sep 17 17:34:54.432110 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:54.432087    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5fdf588c86-cxcfh"]
Sep 17 17:34:54.432263 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:54.432170    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5fdf588c86-cxcfh"
Sep 17 17:34:54.436800 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:54.435269    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 17 17:34:54.436800 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:54.436485    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 17 17:34:54.436800 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:54.436587    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Sep 17 17:34:54.436800 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:54.436763    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 17 17:34:54.437181 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:54.436899    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-8vptn\""
Sep 17 17:34:54.557998 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:54.557973    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/76b84a3e-8b66-432d-b9f3-4aeaf96d262f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kpnbq\" (UID: \"76b84a3e-8b66-432d-b9f3-4aeaf96d262f\") " pod="openshift-insights/insights-runtime-extractor-kpnbq"
Sep 17 17:34:54.558120 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:54.558014    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/76b84a3e-8b66-432d-b9f3-4aeaf96d262f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kpnbq\" (UID: \"76b84a3e-8b66-432d-b9f3-4aeaf96d262f\") " pod="openshift-insights/insights-runtime-extractor-kpnbq"
Sep 17 17:34:54.558166 ip-10-0-134-96 kubenswrapper[2574]: E0917 17:34:54.558144    2574 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 17:34:54.558206 ip-10-0-134-96 kubenswrapper[2574]: E0917 17:34:54.558180    2574 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 17:34:54.558206 ip-10-0-134-96 kubenswrapper[2574]: E0917 17:34:54.558198    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/76b84a3e-8b66-432d-b9f3-4aeaf96d262f-insights-runtime-extractor-tls podName:76b84a3e-8b66-432d-b9f3-4aeaf96d262f nodeName:}" failed. No retries permitted until 2026-09-17 17:34:56.558178993 +0000 UTC m=+51.747052694 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/76b84a3e-8b66-432d-b9f3-4aeaf96d262f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-kpnbq" (UID: "76b84a3e-8b66-432d-b9f3-4aeaf96d262f") : secret "insights-runtime-extractor-tls" not found
Sep 17 17:34:54.558277 ip-10-0-134-96 kubenswrapper[2574]: E0917 17:34:54.558221    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/76b84a3e-8b66-432d-b9f3-4aeaf96d262f-kube-rbac-proxy-cm podName:76b84a3e-8b66-432d-b9f3-4aeaf96d262f nodeName:}" failed. No retries permitted until 2026-09-17 17:34:56.558207885 +0000 UTC m=+51.747081583 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/76b84a3e-8b66-432d-b9f3-4aeaf96d262f-kube-rbac-proxy-cm") pod "insights-runtime-extractor-kpnbq" (UID: "76b84a3e-8b66-432d-b9f3-4aeaf96d262f") : configmap "kube-rbac-proxy" not found
Sep 17 17:34:54.558277 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:54.558239    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/6254338b-9f73-44e2-aa0d-c224d026e0a2-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-5fdf588c86-cxcfh\" (UID: \"6254338b-9f73-44e2-aa0d-c224d026e0a2\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5fdf588c86-cxcfh"
Sep 17 17:34:54.558389 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:54.558370    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8lvsm\" (UniqueName: \"kubernetes.io/projected/6254338b-9f73-44e2-aa0d-c224d026e0a2-kube-api-access-8lvsm\") pod \"managed-serviceaccount-addon-agent-5fdf588c86-cxcfh\" (UID: \"6254338b-9f73-44e2-aa0d-c224d026e0a2\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5fdf588c86-cxcfh"
Sep 17 17:34:54.658928 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:54.658902    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8lvsm\" (UniqueName: \"kubernetes.io/projected/6254338b-9f73-44e2-aa0d-c224d026e0a2-kube-api-access-8lvsm\") pod \"managed-serviceaccount-addon-agent-5fdf588c86-cxcfh\" (UID: \"6254338b-9f73-44e2-aa0d-c224d026e0a2\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5fdf588c86-cxcfh"
Sep 17 17:34:54.659074 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:54.659053    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/6254338b-9f73-44e2-aa0d-c224d026e0a2-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-5fdf588c86-cxcfh\" (UID: \"6254338b-9f73-44e2-aa0d-c224d026e0a2\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5fdf588c86-cxcfh"
Sep 17 17:34:54.661260 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:54.661233    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/6254338b-9f73-44e2-aa0d-c224d026e0a2-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-5fdf588c86-cxcfh\" (UID: \"6254338b-9f73-44e2-aa0d-c224d026e0a2\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5fdf588c86-cxcfh"
Sep 17 17:34:54.667384 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:54.667337    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8lvsm\" (UniqueName: \"kubernetes.io/projected/6254338b-9f73-44e2-aa0d-c224d026e0a2-kube-api-access-8lvsm\") pod \"managed-serviceaccount-addon-agent-5fdf588c86-cxcfh\" (UID: \"6254338b-9f73-44e2-aa0d-c224d026e0a2\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5fdf588c86-cxcfh"
Sep 17 17:34:54.749577 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:54.749551    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5fdf588c86-cxcfh"
Sep 17 17:34:54.859112 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:54.859079    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j4ds_d3f553e8-b416-4d7d-bbef-991a0404b287/konnectivity-agent/0.log" limit=1024
Sep 17 17:34:54.936288 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:54.936183    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j4ds_d3f553e8-b416-4d7d-bbef-991a0404b287/konnectivity-agent/0.log" limit=1024
Sep 17 17:34:55.037607 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:55.037579    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5fdf588c86-cxcfh"]
Sep 17 17:34:55.040852 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:34:55.040829    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6254338b_9f73_44e2_aa0d_c224d026e0a2.slice/crio-da161ae0e9f9d84959cbacb47362b5834fe833f0d3bb553963818ac1b02e3467 WatchSource:0}: Error finding container da161ae0e9f9d84959cbacb47362b5834fe833f0d3bb553963818ac1b02e3467: Status 404 returned error can't find the container with id da161ae0e9f9d84959cbacb47362b5834fe833f0d3bb553963818ac1b02e3467
Sep 17 17:34:55.309504 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:55.309467    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-7d5q6" event={"ID":"38d5d165-e37b-4a2f-9c2a-67cbb3267248","Type":"ContainerStarted","Data":"8857384ad380263f4ffd8982c1e3835d227c0f730f88f50917ea362a01234323"}
Sep 17 17:34:55.309683 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:55.309527    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-7d5q6" event={"ID":"38d5d165-e37b-4a2f-9c2a-67cbb3267248","Type":"ContainerStarted","Data":"6df4b01ca4a8b46a03efc22ba145668a384071b3ec2fe1333e2cc8472859b5cc"}
Sep 17 17:34:55.310538 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:55.310518    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5fdf588c86-cxcfh" event={"ID":"6254338b-9f73-44e2-aa0d-c224d026e0a2","Type":"ContainerStarted","Data":"da161ae0e9f9d84959cbacb47362b5834fe833f0d3bb553963818ac1b02e3467"}
Sep 17 17:34:55.329415 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:55.329371    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-7d5q6" podStartSLOduration=1.70327944 podStartE2EDuration="3.329356698s" totalImagesPullingTime="1.626077258s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:34:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:34:55.328945294 +0000 UTC m=+50.517819002" watchObservedRunningTime="2026-09-17 17:34:55.329356698 +0000 UTC m=+50.518230405"
Sep 17 17:34:55.367438 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:55.367414    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-lf4tf_31495fae-ecfa-4263-a3db-702f75935118/dns-node-resolver/0.log"
Sep 17 17:34:55.427853 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:55.427828    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j4ds_d3f553e8-b416-4d7d-bbef-991a0404b287/konnectivity-agent/0.log" limit=1024
Sep 17 17:34:55.466317 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:55.466292    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d4f6642f-b0fd-4434-a318-530eac3af0a0-metrics-tls\") pod \"dns-default-tnkgs\" (UID: \"d4f6642f-b0fd-4434-a318-530eac3af0a0\") " pod="openshift-dns/dns-default-tnkgs"
Sep 17 17:34:55.466435 ip-10-0-134-96 kubenswrapper[2574]: E0917 17:34:55.466412    2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 17:34:55.466479 ip-10-0-134-96 kubenswrapper[2574]: E0917 17:34:55.466458    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d4f6642f-b0fd-4434-a318-530eac3af0a0-metrics-tls podName:d4f6642f-b0fd-4434-a318-530eac3af0a0 nodeName:}" failed. No retries permitted until 2026-09-17 17:35:11.466445506 +0000 UTC m=+66.655319190 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d4f6642f-b0fd-4434-a318-530eac3af0a0-metrics-tls") pod "dns-default-tnkgs" (UID: "d4f6642f-b0fd-4434-a318-530eac3af0a0") : secret "dns-default-metrics-tls" not found
Sep 17 17:34:55.510257 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:55.510223    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j4ds_d3f553e8-b416-4d7d-bbef-991a0404b287/konnectivity-agent/0.log" limit=1024
Sep 17 17:34:56.175385 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:56.175355    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j4ds_d3f553e8-b416-4d7d-bbef-991a0404b287/konnectivity-agent/0.log" limit=1024
Sep 17 17:34:56.574992 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:56.574958    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/76b84a3e-8b66-432d-b9f3-4aeaf96d262f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kpnbq\" (UID: \"76b84a3e-8b66-432d-b9f3-4aeaf96d262f\") " pod="openshift-insights/insights-runtime-extractor-kpnbq"
Sep 17 17:34:56.574992 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:56.575002    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/76b84a3e-8b66-432d-b9f3-4aeaf96d262f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kpnbq\" (UID: \"76b84a3e-8b66-432d-b9f3-4aeaf96d262f\") " pod="openshift-insights/insights-runtime-extractor-kpnbq"
Sep 17 17:34:56.575472 ip-10-0-134-96 kubenswrapper[2574]: E0917 17:34:56.575105    2574 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 17:34:56.575472 ip-10-0-134-96 kubenswrapper[2574]: E0917 17:34:56.575168    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/76b84a3e-8b66-432d-b9f3-4aeaf96d262f-kube-rbac-proxy-cm podName:76b84a3e-8b66-432d-b9f3-4aeaf96d262f nodeName:}" failed. No retries permitted until 2026-09-17 17:35:00.575151166 +0000 UTC m=+55.764024857 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/76b84a3e-8b66-432d-b9f3-4aeaf96d262f-kube-rbac-proxy-cm") pod "insights-runtime-extractor-kpnbq" (UID: "76b84a3e-8b66-432d-b9f3-4aeaf96d262f") : configmap "kube-rbac-proxy" not found
Sep 17 17:34:56.575472 ip-10-0-134-96 kubenswrapper[2574]: E0917 17:34:56.575212    2574 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 17:34:56.575472 ip-10-0-134-96 kubenswrapper[2574]: E0917 17:34:56.575283    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/76b84a3e-8b66-432d-b9f3-4aeaf96d262f-insights-runtime-extractor-tls podName:76b84a3e-8b66-432d-b9f3-4aeaf96d262f nodeName:}" failed. No retries permitted until 2026-09-17 17:35:00.575264927 +0000 UTC m=+55.764138632 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/76b84a3e-8b66-432d-b9f3-4aeaf96d262f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-kpnbq" (UID: "76b84a3e-8b66-432d-b9f3-4aeaf96d262f") : secret "insights-runtime-extractor-tls" not found
Sep 17 17:34:56.766605 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:56.766578    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-t79st_67b5a7ce-5a80-4224-9f54-86ac40fa136b/node-ca/0.log"
Sep 17 17:34:57.165518 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:57.165494    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-2sz2c_0f867410-d783-47d8-91aa-564f7e5cecc8/serve-healthcheck-canary/0.log"
Sep 17 17:34:57.767119 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:57.767090    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-76794cbd6c-7d5q6_38d5d165-e37b-4a2f-9c2a-67cbb3267248/migrator/0.log"
Sep 17 17:34:57.966654 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:57.966582    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-76794cbd6c-7d5q6_38d5d165-e37b-4a2f-9c2a-67cbb3267248/graceful-termination/0.log"
Sep 17 17:34:58.166172 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:58.166141    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-55889f7b47-ns2j6_88497892-29d0-4b92-803b-2ce9ebebf402/kube-storage-version-migrator-operator/0.log"
Sep 17 17:34:58.322115 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:58.322079    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5fdf588c86-cxcfh" event={"ID":"6254338b-9f73-44e2-aa0d-c224d026e0a2","Type":"ContainerStarted","Data":"89f7509b2c789272e7e48bb16f954d4b0cdb9008dd831d30033621145dfdfb31"}
Sep 17 17:34:58.338749 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:58.338697    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5fdf588c86-cxcfh" podStartSLOduration=1.8109042180000001 podStartE2EDuration="4.338681761s" totalImagesPullingTime="2.527777543s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:34:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:34:58.337831274 +0000 UTC m=+53.526704982" watchObservedRunningTime="2026-09-17 17:34:58.338681761 +0000 UTC m=+53.527555506"
Sep 17 17:34:59.086817 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:59.086771    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j4ds_d3f553e8-b416-4d7d-bbef-991a0404b287/konnectivity-agent/0.log" limit=1024
Sep 17 17:34:59.975238 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:34:59.975191    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j4ds_d3f553e8-b416-4d7d-bbef-991a0404b287/konnectivity-agent/0.log" limit=1024
Sep 17 17:35:00.075473 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:00.075425    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j4ds_d3f553e8-b416-4d7d-bbef-991a0404b287/konnectivity-agent/0.log" limit=1024
Sep 17 17:35:00.181940 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:00.181905    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j4ds_d3f553e8-b416-4d7d-bbef-991a0404b287/konnectivity-agent/0.log" limit=1024
Sep 17 17:35:00.285298 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:00.285263    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j4ds_d3f553e8-b416-4d7d-bbef-991a0404b287/konnectivity-agent/0.log" limit=1024
Sep 17 17:35:00.382931 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:00.382899    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j4ds_d3f553e8-b416-4d7d-bbef-991a0404b287/konnectivity-agent/0.log" limit=1024
Sep 17 17:35:00.610234 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:00.610157    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/76b84a3e-8b66-432d-b9f3-4aeaf96d262f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kpnbq\" (UID: \"76b84a3e-8b66-432d-b9f3-4aeaf96d262f\") " pod="openshift-insights/insights-runtime-extractor-kpnbq"
Sep 17 17:35:00.610234 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:00.610195    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/76b84a3e-8b66-432d-b9f3-4aeaf96d262f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kpnbq\" (UID: \"76b84a3e-8b66-432d-b9f3-4aeaf96d262f\") " pod="openshift-insights/insights-runtime-extractor-kpnbq"
Sep 17 17:35:00.610421 ip-10-0-134-96 kubenswrapper[2574]: E0917 17:35:00.610273    2574 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 17:35:00.610421 ip-10-0-134-96 kubenswrapper[2574]: E0917 17:35:00.610316    2574 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 17:35:00.610421 ip-10-0-134-96 kubenswrapper[2574]: E0917 17:35:00.610338    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/76b84a3e-8b66-432d-b9f3-4aeaf96d262f-kube-rbac-proxy-cm podName:76b84a3e-8b66-432d-b9f3-4aeaf96d262f nodeName:}" failed. No retries permitted until 2026-09-17 17:35:08.610321465 +0000 UTC m=+63.799195150 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/76b84a3e-8b66-432d-b9f3-4aeaf96d262f-kube-rbac-proxy-cm") pod "insights-runtime-extractor-kpnbq" (UID: "76b84a3e-8b66-432d-b9f3-4aeaf96d262f") : configmap "kube-rbac-proxy" not found
Sep 17 17:35:00.610421 ip-10-0-134-96 kubenswrapper[2574]: E0917 17:35:00.610356    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/76b84a3e-8b66-432d-b9f3-4aeaf96d262f-insights-runtime-extractor-tls podName:76b84a3e-8b66-432d-b9f3-4aeaf96d262f nodeName:}" failed. No retries permitted until 2026-09-17 17:35:08.61034742 +0000 UTC m=+63.799221108 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/76b84a3e-8b66-432d-b9f3-4aeaf96d262f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-kpnbq" (UID: "76b84a3e-8b66-432d-b9f3-4aeaf96d262f") : secret "insights-runtime-extractor-tls" not found
Sep 17 17:35:00.893499 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:00.893425    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j4ds_d3f553e8-b416-4d7d-bbef-991a0404b287/konnectivity-agent/0.log" limit=1024
Sep 17 17:35:01.473926 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:01.473897    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j4ds_d3f553e8-b416-4d7d-bbef-991a0404b287/konnectivity-agent/0.log" limit=1024
Sep 17 17:35:01.616833 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:01.616805    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/b3d9c617-b694-40c2-98ce-3772a892df3c-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-5ct2x\" (UID: \"b3d9c617-b694-40c2-98ce-3772a892df3c\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-5ct2x"
Sep 17 17:35:01.616990 ip-10-0-134-96 kubenswrapper[2574]: E0917 17:35:01.616902    2574 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 17 17:35:01.616990 ip-10-0-134-96 kubenswrapper[2574]: E0917 17:35:01.616955    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b3d9c617-b694-40c2-98ce-3772a892df3c-cluster-monitoring-operator-tls podName:b3d9c617-b694-40c2-98ce-3772a892df3c nodeName:}" failed. No retries permitted until 2026-09-17 17:35:17.616940647 +0000 UTC m=+72.805814331 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/b3d9c617-b694-40c2-98ce-3772a892df3c-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-5ct2x" (UID: "b3d9c617-b694-40c2-98ce-3772a892df3c") : secret "cluster-monitoring-operator-tls" not found
Sep 17 17:35:02.047057 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:02.047004    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j4ds_d3f553e8-b416-4d7d-bbef-991a0404b287/konnectivity-agent/0.log" limit=1024
Sep 17 17:35:02.329326 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:02.329250    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j4ds_d3f553e8-b416-4d7d-bbef-991a0404b287/konnectivity-agent/0.log" limit=1024
Sep 17 17:35:02.653323 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:02.653248    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j4ds_d3f553e8-b416-4d7d-bbef-991a0404b287/konnectivity-agent/0.log" limit=1024
Sep 17 17:35:02.883944 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:02.883913    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j4ds_d3f553e8-b416-4d7d-bbef-991a0404b287/konnectivity-agent/0.log" limit=1024
Sep 17 17:35:03.252864 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:03.252804    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j4ds_d3f553e8-b416-4d7d-bbef-991a0404b287/konnectivity-agent/0.log" limit=1024
Sep 17 17:35:03.549614 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:03.549543    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j4ds_d3f553e8-b416-4d7d-bbef-991a0404b287/konnectivity-agent/0.log" limit=1024
Sep 17 17:35:03.653478 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:03.653446    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j4ds_d3f553e8-b416-4d7d-bbef-991a0404b287/konnectivity-agent/0.log" limit=1024
Sep 17 17:35:03.899574 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:03.899503    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j4ds_d3f553e8-b416-4d7d-bbef-991a0404b287/konnectivity-agent/0.log" limit=1024
Sep 17 17:35:04.333420 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:04.333389    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j4ds_d3f553e8-b416-4d7d-bbef-991a0404b287/konnectivity-agent/0.log" limit=1024
Sep 17 17:35:04.430991 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:04.430963    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j4ds_d3f553e8-b416-4d7d-bbef-991a0404b287/konnectivity-agent/0.log" limit=1024
Sep 17 17:35:04.508890 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:04.508858    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j4ds_d3f553e8-b416-4d7d-bbef-991a0404b287/konnectivity-agent/0.log" limit=1024
Sep 17 17:35:04.857589 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:04.857557    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j4ds_d3f553e8-b416-4d7d-bbef-991a0404b287/konnectivity-agent/0.log" limit=1024
Sep 17 17:35:05.056681 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:05.056652    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j4ds_d3f553e8-b416-4d7d-bbef-991a0404b287/konnectivity-agent/0.log" limit=1024
Sep 17 17:35:05.232347 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:05.232285    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-45dp8"
Sep 17 17:35:05.304445 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:05.304415    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j4ds_d3f553e8-b416-4d7d-bbef-991a0404b287/konnectivity-agent/0.log" limit=1024
Sep 17 17:35:06.245659 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:06.245628    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j4ds_d3f553e8-b416-4d7d-bbef-991a0404b287/konnectivity-agent/0.log" limit=1024
Sep 17 17:35:06.348916 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:06.348890    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j4ds_d3f553e8-b416-4d7d-bbef-991a0404b287/konnectivity-agent/0.log" limit=1024
Sep 17 17:35:06.470766 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:06.470733    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j4ds_d3f553e8-b416-4d7d-bbef-991a0404b287/konnectivity-agent/0.log" limit=1024
Sep 17 17:35:06.654342 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:06.654273    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j4ds_d3f553e8-b416-4d7d-bbef-991a0404b287/konnectivity-agent/0.log" limit=1024
Sep 17 17:35:06.886564 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:06.886534    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j4ds_d3f553e8-b416-4d7d-bbef-991a0404b287/konnectivity-agent/0.log" limit=1024
Sep 17 17:35:07.069971 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:07.069946    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j4ds_d3f553e8-b416-4d7d-bbef-991a0404b287/konnectivity-agent/0.log" limit=1024
Sep 17 17:35:08.056120 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:08.056087    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j4ds_d3f553e8-b416-4d7d-bbef-991a0404b287/konnectivity-agent/0.log" limit=1024
Sep 17 17:35:08.226747 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:08.226717    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j4ds_d3f553e8-b416-4d7d-bbef-991a0404b287/konnectivity-agent/0.log" limit=1024
Sep 17 17:35:08.342245 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:08.342173    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j4ds_d3f553e8-b416-4d7d-bbef-991a0404b287/konnectivity-agent/0.log" limit=1024
Sep 17 17:35:08.507544 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:08.507511    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j4ds_d3f553e8-b416-4d7d-bbef-991a0404b287/konnectivity-agent/0.log" limit=1024
Sep 17 17:35:08.645423 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:08.645348    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j4ds_d3f553e8-b416-4d7d-bbef-991a0404b287/konnectivity-agent/0.log" limit=1024
Sep 17 17:35:08.675548 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:08.675524    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/76b84a3e-8b66-432d-b9f3-4aeaf96d262f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kpnbq\" (UID: \"76b84a3e-8b66-432d-b9f3-4aeaf96d262f\") " pod="openshift-insights/insights-runtime-extractor-kpnbq"
Sep 17 17:35:08.675671 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:08.675561    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/76b84a3e-8b66-432d-b9f3-4aeaf96d262f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kpnbq\" (UID: \"76b84a3e-8b66-432d-b9f3-4aeaf96d262f\") " pod="openshift-insights/insights-runtime-extractor-kpnbq"
Sep 17 17:35:08.675671 ip-10-0-134-96 kubenswrapper[2574]: E0917 17:35:08.675651    2574 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 17:35:08.675737 ip-10-0-134-96 kubenswrapper[2574]: E0917 17:35:08.675711    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/76b84a3e-8b66-432d-b9f3-4aeaf96d262f-insights-runtime-extractor-tls podName:76b84a3e-8b66-432d-b9f3-4aeaf96d262f nodeName:}" failed. No retries permitted until 2026-09-17 17:35:24.675689211 +0000 UTC m=+79.864562902 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/76b84a3e-8b66-432d-b9f3-4aeaf96d262f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-kpnbq" (UID: "76b84a3e-8b66-432d-b9f3-4aeaf96d262f") : secret "insights-runtime-extractor-tls" not found
Sep 17 17:35:08.676099 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:08.676081    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/76b84a3e-8b66-432d-b9f3-4aeaf96d262f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kpnbq\" (UID: \"76b84a3e-8b66-432d-b9f3-4aeaf96d262f\") " pod="openshift-insights/insights-runtime-extractor-kpnbq"
Sep 17 17:35:08.750398 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:08.750366    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j4ds_d3f553e8-b416-4d7d-bbef-991a0404b287/konnectivity-agent/0.log" limit=1024
Sep 17 17:35:08.853173 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:08.853127    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j4ds_d3f553e8-b416-4d7d-bbef-991a0404b287/konnectivity-agent/0.log" limit=1024
Sep 17 17:35:08.959134 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:08.959069    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j4ds_d3f553e8-b416-4d7d-bbef-991a0404b287/konnectivity-agent/0.log" limit=1024
Sep 17 17:35:09.051949 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:09.051923    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j4ds_d3f553e8-b416-4d7d-bbef-991a0404b287/konnectivity-agent/0.log" limit=1024
Sep 17 17:35:09.266160 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:09.266130    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j4ds_d3f553e8-b416-4d7d-bbef-991a0404b287/konnectivity-agent/0.log" limit=1024
Sep 17 17:35:09.393231 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:09.393203    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j4ds_d3f553e8-b416-4d7d-bbef-991a0404b287/konnectivity-agent/0.log" limit=1024
Sep 17 17:35:09.886322 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:09.886292    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j4ds_d3f553e8-b416-4d7d-bbef-991a0404b287/konnectivity-agent/0.log" limit=1024
Sep 17 17:35:10.268860 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:10.268832    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j4ds_d3f553e8-b416-4d7d-bbef-991a0404b287/konnectivity-agent/0.log" limit=1024
Sep 17 17:35:10.397469 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:10.397435    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j4ds_d3f553e8-b416-4d7d-bbef-991a0404b287/konnectivity-agent/0.log" limit=1024
Sep 17 17:35:10.488863 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:10.488831    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j4ds_d3f553e8-b416-4d7d-bbef-991a0404b287/konnectivity-agent/0.log" limit=1024
Sep 17 17:35:10.621979 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:10.621899    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j4ds_d3f553e8-b416-4d7d-bbef-991a0404b287/konnectivity-agent/0.log" limit=1024
Sep 17 17:35:10.724279 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:10.724243    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j4ds_d3f553e8-b416-4d7d-bbef-991a0404b287/konnectivity-agent/0.log" limit=1024
Sep 17 17:35:11.201832 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:11.201801    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j4ds_d3f553e8-b416-4d7d-bbef-991a0404b287/konnectivity-agent/0.log" limit=1024
Sep 17 17:35:11.357192 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:11.357163    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j4ds_d3f553e8-b416-4d7d-bbef-991a0404b287/konnectivity-agent/0.log" limit=1024
Sep 17 17:35:11.497177 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:11.497150    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d4f6642f-b0fd-4434-a318-530eac3af0a0-metrics-tls\") pod \"dns-default-tnkgs\" (UID: \"d4f6642f-b0fd-4434-a318-530eac3af0a0\") " pod="openshift-dns/dns-default-tnkgs"
Sep 17 17:35:11.499506 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:11.499478    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d4f6642f-b0fd-4434-a318-530eac3af0a0-metrics-tls\") pod \"dns-default-tnkgs\" (UID: \"d4f6642f-b0fd-4434-a318-530eac3af0a0\") " pod="openshift-dns/dns-default-tnkgs"
Sep 17 17:35:11.698149 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:11.698105    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fadd68fa-727c-4581-bed0-5ab8b73b9982-original-pull-secret\") pod \"global-pull-secret-syncer-sv2qq\" (UID: \"fadd68fa-727c-4581-bed0-5ab8b73b9982\") " pod="kube-system/global-pull-secret-syncer-sv2qq"
Sep 17 17:35:11.698328 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:11.698288    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c18eb096-7089-426a-957b-3dc1a2d7e837-metrics-certs\") pod \"network-metrics-daemon-4tpm5\" (UID: \"c18eb096-7089-426a-957b-3dc1a2d7e837\") " pod="openshift-multus/network-metrics-daemon-4tpm5"
Sep 17 17:35:11.700859 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:11.700842    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 17 17:35:11.701816 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:11.701800    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 17 17:35:11.711194 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:11.711175    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fadd68fa-727c-4581-bed0-5ab8b73b9982-original-pull-secret\") pod \"global-pull-secret-syncer-sv2qq\" (UID: \"fadd68fa-727c-4581-bed0-5ab8b73b9982\") " pod="kube-system/global-pull-secret-syncer-sv2qq"
Sep 17 17:35:11.711278 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:11.711239    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c18eb096-7089-426a-957b-3dc1a2d7e837-metrics-certs\") pod \"network-metrics-daemon-4tpm5\" (UID: \"c18eb096-7089-426a-957b-3dc1a2d7e837\") " pod="openshift-multus/network-metrics-daemon-4tpm5"
Sep 17 17:35:11.761077 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:11.761015    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-qcgkp\""
Sep 17 17:35:11.763243 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:11.763229    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-bd5nq\""
Sep 17 17:35:11.768169 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:11.768153    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sv2qq"
Sep 17 17:35:11.769223 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:11.769210    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4tpm5"
Sep 17 17:35:11.771606 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:11.771590    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-tnkgs"
Sep 17 17:35:11.799687 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:11.799643    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2lk56\" (UniqueName: \"kubernetes.io/projected/61c43e2e-f36c-401d-a10c-ff5f88fab086-kube-api-access-2lk56\") pod \"network-check-target-j9d2w\" (UID: \"61c43e2e-f36c-401d-a10c-ff5f88fab086\") " pod="openshift-network-diagnostics/network-check-target-j9d2w"
Sep 17 17:35:11.802684 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:11.802651    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2lk56\" (UniqueName: \"kubernetes.io/projected/61c43e2e-f36c-401d-a10c-ff5f88fab086-kube-api-access-2lk56\") pod \"network-check-target-j9d2w\" (UID: \"61c43e2e-f36c-401d-a10c-ff5f88fab086\") " pod="openshift-network-diagnostics/network-check-target-j9d2w"
Sep 17 17:35:12.043011 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:12.042942    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j4ds_d3f553e8-b416-4d7d-bbef-991a0404b287/konnectivity-agent/0.log" limit=1024
Sep 17 17:35:12.044930 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:12.044904    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-sv2qq"]
Sep 17 17:35:12.047736 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:35:12.047715    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfadd68fa_727c_4581_bed0_5ab8b73b9982.slice/crio-4ae0efcb39257fdf9755366e231041a9d9b35d482d5971d24b6e651969acd9a3 WatchSource:0}: Error finding container 4ae0efcb39257fdf9755366e231041a9d9b35d482d5971d24b6e651969acd9a3: Status 404 returned error can't find the container with id 4ae0efcb39257fdf9755366e231041a9d9b35d482d5971d24b6e651969acd9a3
Sep 17 17:35:12.055138 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:12.055016    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-4tpm5"]
Sep 17 17:35:12.056058 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:12.056021    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-tnkgs"]
Sep 17 17:35:12.059141 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:35:12.059114    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc18eb096_7089_426a_957b_3dc1a2d7e837.slice/crio-15e288aee82daceadf98d9d3b86a27a6eeec317ff7ff515cbc5419e5543a38af WatchSource:0}: Error finding container 15e288aee82daceadf98d9d3b86a27a6eeec317ff7ff515cbc5419e5543a38af: Status 404 returned error can't find the container with id 15e288aee82daceadf98d9d3b86a27a6eeec317ff7ff515cbc5419e5543a38af
Sep 17 17:35:12.059708 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:35:12.059688    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4f6642f_b0fd_4434_a318_530eac3af0a0.slice/crio-1861604c590aac7c85a523a4270a43ad1a6619805473bd9d52a3e0a5557ec07c WatchSource:0}: Error finding container 1861604c590aac7c85a523a4270a43ad1a6619805473bd9d52a3e0a5557ec07c: Status 404 returned error can't find the container with id 1861604c590aac7c85a523a4270a43ad1a6619805473bd9d52a3e0a5557ec07c
Sep 17 17:35:12.066234 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:12.066215    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-crvff\""
Sep 17 17:35:12.074355 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:12.074334    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j9d2w"
Sep 17 17:35:12.343439 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:12.343368    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-j9d2w"]
Sep 17 17:35:12.346249 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:35:12.346223    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61c43e2e_f36c_401d_a10c_ff5f88fab086.slice/crio-656a1b796b23870a34301e4749762008d4b74c50c68d6dd5d38fd5098b250ee0 WatchSource:0}: Error finding container 656a1b796b23870a34301e4749762008d4b74c50c68d6dd5d38fd5098b250ee0: Status 404 returned error can't find the container with id 656a1b796b23870a34301e4749762008d4b74c50c68d6dd5d38fd5098b250ee0
Sep 17 17:35:12.370698 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:12.370670    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-j9d2w" event={"ID":"61c43e2e-f36c-401d-a10c-ff5f88fab086","Type":"ContainerStarted","Data":"656a1b796b23870a34301e4749762008d4b74c50c68d6dd5d38fd5098b250ee0"}
Sep 17 17:35:12.371746 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:12.371727    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-4tpm5" event={"ID":"c18eb096-7089-426a-957b-3dc1a2d7e837","Type":"ContainerStarted","Data":"15e288aee82daceadf98d9d3b86a27a6eeec317ff7ff515cbc5419e5543a38af"}
Sep 17 17:35:12.372755 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:12.372735    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-tnkgs" event={"ID":"d4f6642f-b0fd-4434-a318-530eac3af0a0","Type":"ContainerStarted","Data":"1861604c590aac7c85a523a4270a43ad1a6619805473bd9d52a3e0a5557ec07c"}
Sep 17 17:35:12.374192 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:12.374170    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-sv2qq" event={"ID":"fadd68fa-727c-4581-bed0-5ab8b73b9982","Type":"ContainerStarted","Data":"4ae0efcb39257fdf9755366e231041a9d9b35d482d5971d24b6e651969acd9a3"}
Sep 17 17:35:13.380902 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:13.380808    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-4tpm5" event={"ID":"c18eb096-7089-426a-957b-3dc1a2d7e837","Type":"ContainerStarted","Data":"15ea15f05dc1816780eac32f92f3b580c048b98504702e31a921a95a1d851591"}
Sep 17 17:35:13.380902 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:13.380882    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-4tpm5" event={"ID":"c18eb096-7089-426a-957b-3dc1a2d7e837","Type":"ContainerStarted","Data":"16a176ccc2a6a0350124a30e515f140e4558aca788a8e42f14e41e03e15d22c4"}
Sep 17 17:35:13.382789 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:13.382756    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-j9d2w" event={"ID":"61c43e2e-f36c-401d-a10c-ff5f88fab086","Type":"ContainerStarted","Data":"80023cf22877e9de9bb9be9f2ac0791daecf4979dcad07d8c8cf3e3bed5349ba"}
Sep 17 17:35:13.382928 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:13.382910    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-j9d2w"
Sep 17 17:35:13.400810 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:13.400764    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-4tpm5" podStartSLOduration=66.439531197 podStartE2EDuration="1m7.400746002s" totalImagesPullingTime="961.214805ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:34:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:35:13.39943682 +0000 UTC m=+68.588310527" watchObservedRunningTime="2026-09-17 17:35:13.400746002 +0000 UTC m=+68.589619763"
Sep 17 17:35:13.420601 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:13.420563    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-j9d2w" podStartSLOduration=66.420550131 podStartE2EDuration="1m6.420550131s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:34:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:35:13.420458779 +0000 UTC m=+68.609332488" watchObservedRunningTime="2026-09-17 17:35:13.420550131 +0000 UTC m=+68.609423892"
Sep 17 17:35:13.545389 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:13.545359    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j4ds_d3f553e8-b416-4d7d-bbef-991a0404b287/konnectivity-agent/0.log" limit=1024
Sep 17 17:35:13.616146 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:13.616110    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j4ds_d3f553e8-b416-4d7d-bbef-991a0404b287/konnectivity-agent/0.log" limit=1024
Sep 17 17:35:14.393999 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:14.393959    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-tnkgs" event={"ID":"d4f6642f-b0fd-4434-a318-530eac3af0a0","Type":"ContainerStarted","Data":"10fd3e65e3b83cf7f75787a0f8fdb619fe0e8322ea5ab502ca7aeac1f4419430"}
Sep 17 17:35:14.394449 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:14.394073    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-tnkgs" event={"ID":"d4f6642f-b0fd-4434-a318-530eac3af0a0","Type":"ContainerStarted","Data":"a79a8d3ae4e89d3f6b82b4611b999d49bd381632b8cae66f6dd54e7873c7d68c"}
Sep 17 17:35:14.394449 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:14.394245    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-tnkgs"
Sep 17 17:35:14.416869 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:14.416821    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-tnkgs" podStartSLOduration=33.721949461 podStartE2EDuration="35.416800549s" totalImagesPullingTime="1.694851088s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:34:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:35:14.415755659 +0000 UTC m=+69.604629367" watchObservedRunningTime="2026-09-17 17:35:14.416800549 +0000 UTC m=+69.605674258"
Sep 17 17:35:16.402750 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:16.402714    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-sv2qq" event={"ID":"fadd68fa-727c-4581-bed0-5ab8b73b9982","Type":"ContainerStarted","Data":"6f0cb24dc6998df5bbb1ff20fdbed4524d6512b770015efbaaf8a400c176c682"}
Sep 17 17:35:16.418651 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:16.418600    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-sv2qq" podStartSLOduration=65.596282132 podStartE2EDuration="1m9.418585631s" totalImagesPullingTime="3.822303499s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:34:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:35:16.418113449 +0000 UTC m=+71.606987157" watchObservedRunningTime="2026-09-17 17:35:16.418585631 +0000 UTC m=+71.607459486"
Sep 17 17:35:16.594628 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:16.594598    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j4ds_d3f553e8-b416-4d7d-bbef-991a0404b287/konnectivity-agent/0.log" limit=1024
Sep 17 17:35:17.643166 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:17.643060    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/b3d9c617-b694-40c2-98ce-3772a892df3c-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-5ct2x\" (UID: \"b3d9c617-b694-40c2-98ce-3772a892df3c\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-5ct2x"
Sep 17 17:35:17.645437 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:17.645414    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/b3d9c617-b694-40c2-98ce-3772a892df3c-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-5ct2x\" (UID: \"b3d9c617-b694-40c2-98ce-3772a892df3c\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-5ct2x"
Sep 17 17:35:17.897545 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:17.897474    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-zbmkl\""
Sep 17 17:35:17.905531 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:17.905513    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-5ct2x"
Sep 17 17:35:18.198939 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:18.198870    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-5ct2x"]
Sep 17 17:35:18.201720 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:35:18.201692    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3d9c617_b694_40c2_98ce_3772a892df3c.slice/crio-ec4d9ed6a42cf730657cec73e0551886eab3650878785447e2520cc6a48e0b64 WatchSource:0}: Error finding container ec4d9ed6a42cf730657cec73e0551886eab3650878785447e2520cc6a48e0b64: Status 404 returned error can't find the container with id ec4d9ed6a42cf730657cec73e0551886eab3650878785447e2520cc6a48e0b64
Sep 17 17:35:18.409505 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:18.409475    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-5ct2x" event={"ID":"b3d9c617-b694-40c2-98ce-3772a892df3c","Type":"ContainerStarted","Data":"ec4d9ed6a42cf730657cec73e0551886eab3650878785447e2520cc6a48e0b64"}
Sep 17 17:35:18.471575 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:18.471499    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-58dbcf67c5-dndpx"]
Sep 17 17:35:18.476444 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:18.476424    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-58dbcf67c5-dndpx"
Sep 17 17:35:18.479242 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:18.479216    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 17 17:35:18.479385 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:18.479353    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 17 17:35:18.480395 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:18.480375    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 17 17:35:18.480498 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:18.480452    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-jxpgb\""
Sep 17 17:35:18.493965 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:18.490898    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 17 17:35:18.501660 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:18.501639    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-58dbcf67c5-dndpx"]
Sep 17 17:35:18.547579 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:18.547553    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j4ds_d3f553e8-b416-4d7d-bbef-991a0404b287/konnectivity-agent/0.log" limit=1024
Sep 17 17:35:18.550640 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:18.550624    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1e86520f-fe34-47cb-a4fc-2c01d1f9c9be-bound-sa-token\") pod \"image-registry-58dbcf67c5-dndpx\" (UID: \"1e86520f-fe34-47cb-a4fc-2c01d1f9c9be\") " pod="openshift-image-registry/image-registry-58dbcf67c5-dndpx"
Sep 17 17:35:18.550700 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:18.550653    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1e86520f-fe34-47cb-a4fc-2c01d1f9c9be-registry-certificates\") pod \"image-registry-58dbcf67c5-dndpx\" (UID: \"1e86520f-fe34-47cb-a4fc-2c01d1f9c9be\") " pod="openshift-image-registry/image-registry-58dbcf67c5-dndpx"
Sep 17 17:35:18.550700 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:18.550676    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h8mml\" (UniqueName: \"kubernetes.io/projected/1e86520f-fe34-47cb-a4fc-2c01d1f9c9be-kube-api-access-h8mml\") pod \"image-registry-58dbcf67c5-dndpx\" (UID: \"1e86520f-fe34-47cb-a4fc-2c01d1f9c9be\") " pod="openshift-image-registry/image-registry-58dbcf67c5-dndpx"
Sep 17 17:35:18.550700 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:18.550693    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1e86520f-fe34-47cb-a4fc-2c01d1f9c9be-registry-tls\") pod \"image-registry-58dbcf67c5-dndpx\" (UID: \"1e86520f-fe34-47cb-a4fc-2c01d1f9c9be\") " pod="openshift-image-registry/image-registry-58dbcf67c5-dndpx"
Sep 17 17:35:18.550805 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:18.550744    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1e86520f-fe34-47cb-a4fc-2c01d1f9c9be-ca-trust-extracted\") pod \"image-registry-58dbcf67c5-dndpx\" (UID: \"1e86520f-fe34-47cb-a4fc-2c01d1f9c9be\") " pod="openshift-image-registry/image-registry-58dbcf67c5-dndpx"
Sep 17 17:35:18.550805 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:18.550772    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1e86520f-fe34-47cb-a4fc-2c01d1f9c9be-installation-pull-secrets\") pod \"image-registry-58dbcf67c5-dndpx\" (UID: \"1e86520f-fe34-47cb-a4fc-2c01d1f9c9be\") " pod="openshift-image-registry/image-registry-58dbcf67c5-dndpx"
Sep 17 17:35:18.550805 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:18.550797    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1e86520f-fe34-47cb-a4fc-2c01d1f9c9be-trusted-ca\") pod \"image-registry-58dbcf67c5-dndpx\" (UID: \"1e86520f-fe34-47cb-a4fc-2c01d1f9c9be\") " pod="openshift-image-registry/image-registry-58dbcf67c5-dndpx"
Sep 17 17:35:18.550920 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:18.550864    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1e86520f-fe34-47cb-a4fc-2c01d1f9c9be-image-registry-private-configuration\") pod \"image-registry-58dbcf67c5-dndpx\" (UID: \"1e86520f-fe34-47cb-a4fc-2c01d1f9c9be\") " pod="openshift-image-registry/image-registry-58dbcf67c5-dndpx"
Sep 17 17:35:18.639298 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:18.639269    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j4ds_d3f553e8-b416-4d7d-bbef-991a0404b287/konnectivity-agent/0.log" limit=1024
Sep 17 17:35:18.651517 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:18.651494    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1e86520f-fe34-47cb-a4fc-2c01d1f9c9be-image-registry-private-configuration\") pod \"image-registry-58dbcf67c5-dndpx\" (UID: \"1e86520f-fe34-47cb-a4fc-2c01d1f9c9be\") " pod="openshift-image-registry/image-registry-58dbcf67c5-dndpx"
Sep 17 17:35:18.651868 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:18.651523    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1e86520f-fe34-47cb-a4fc-2c01d1f9c9be-bound-sa-token\") pod \"image-registry-58dbcf67c5-dndpx\" (UID: \"1e86520f-fe34-47cb-a4fc-2c01d1f9c9be\") " pod="openshift-image-registry/image-registry-58dbcf67c5-dndpx"
Sep 17 17:35:18.651868 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:18.651545    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1e86520f-fe34-47cb-a4fc-2c01d1f9c9be-registry-certificates\") pod \"image-registry-58dbcf67c5-dndpx\" (UID: \"1e86520f-fe34-47cb-a4fc-2c01d1f9c9be\") " pod="openshift-image-registry/image-registry-58dbcf67c5-dndpx"
Sep 17 17:35:18.651868 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:18.651588    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h8mml\" (UniqueName: \"kubernetes.io/projected/1e86520f-fe34-47cb-a4fc-2c01d1f9c9be-kube-api-access-h8mml\") pod \"image-registry-58dbcf67c5-dndpx\" (UID: \"1e86520f-fe34-47cb-a4fc-2c01d1f9c9be\") " pod="openshift-image-registry/image-registry-58dbcf67c5-dndpx"
Sep 17 17:35:18.651868 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:18.651613    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1e86520f-fe34-47cb-a4fc-2c01d1f9c9be-registry-tls\") pod \"image-registry-58dbcf67c5-dndpx\" (UID: \"1e86520f-fe34-47cb-a4fc-2c01d1f9c9be\") " pod="openshift-image-registry/image-registry-58dbcf67c5-dndpx"
Sep 17 17:35:18.651868 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:18.651634    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1e86520f-fe34-47cb-a4fc-2c01d1f9c9be-ca-trust-extracted\") pod \"image-registry-58dbcf67c5-dndpx\" (UID: \"1e86520f-fe34-47cb-a4fc-2c01d1f9c9be\") " pod="openshift-image-registry/image-registry-58dbcf67c5-dndpx"
Sep 17 17:35:18.651868 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:18.651657    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1e86520f-fe34-47cb-a4fc-2c01d1f9c9be-installation-pull-secrets\") pod \"image-registry-58dbcf67c5-dndpx\" (UID: \"1e86520f-fe34-47cb-a4fc-2c01d1f9c9be\") " pod="openshift-image-registry/image-registry-58dbcf67c5-dndpx"
Sep 17 17:35:18.651868 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:18.651683    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1e86520f-fe34-47cb-a4fc-2c01d1f9c9be-trusted-ca\") pod \"image-registry-58dbcf67c5-dndpx\" (UID: \"1e86520f-fe34-47cb-a4fc-2c01d1f9c9be\") " pod="openshift-image-registry/image-registry-58dbcf67c5-dndpx"
Sep 17 17:35:18.652176 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:18.652105    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1e86520f-fe34-47cb-a4fc-2c01d1f9c9be-ca-trust-extracted\") pod \"image-registry-58dbcf67c5-dndpx\" (UID: \"1e86520f-fe34-47cb-a4fc-2c01d1f9c9be\") " pod="openshift-image-registry/image-registry-58dbcf67c5-dndpx"
Sep 17 17:35:18.652670 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:18.652644    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1e86520f-fe34-47cb-a4fc-2c01d1f9c9be-registry-certificates\") pod \"image-registry-58dbcf67c5-dndpx\" (UID: \"1e86520f-fe34-47cb-a4fc-2c01d1f9c9be\") " pod="openshift-image-registry/image-registry-58dbcf67c5-dndpx"
Sep 17 17:35:18.652771 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:18.652756    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1e86520f-fe34-47cb-a4fc-2c01d1f9c9be-trusted-ca\") pod \"image-registry-58dbcf67c5-dndpx\" (UID: \"1e86520f-fe34-47cb-a4fc-2c01d1f9c9be\") " pod="openshift-image-registry/image-registry-58dbcf67c5-dndpx"
Sep 17 17:35:18.654130 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:18.654111    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1e86520f-fe34-47cb-a4fc-2c01d1f9c9be-registry-tls\") pod \"image-registry-58dbcf67c5-dndpx\" (UID: \"1e86520f-fe34-47cb-a4fc-2c01d1f9c9be\") " pod="openshift-image-registry/image-registry-58dbcf67c5-dndpx"
Sep 17 17:35:18.654206 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:18.654179    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1e86520f-fe34-47cb-a4fc-2c01d1f9c9be-installation-pull-secrets\") pod \"image-registry-58dbcf67c5-dndpx\" (UID: \"1e86520f-fe34-47cb-a4fc-2c01d1f9c9be\") " pod="openshift-image-registry/image-registry-58dbcf67c5-dndpx"
Sep 17 17:35:18.654253 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:18.654199    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1e86520f-fe34-47cb-a4fc-2c01d1f9c9be-image-registry-private-configuration\") pod \"image-registry-58dbcf67c5-dndpx\" (UID: \"1e86520f-fe34-47cb-a4fc-2c01d1f9c9be\") " pod="openshift-image-registry/image-registry-58dbcf67c5-dndpx"
Sep 17 17:35:18.672839 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:18.672813    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h8mml\" (UniqueName: \"kubernetes.io/projected/1e86520f-fe34-47cb-a4fc-2c01d1f9c9be-kube-api-access-h8mml\") pod \"image-registry-58dbcf67c5-dndpx\" (UID: \"1e86520f-fe34-47cb-a4fc-2c01d1f9c9be\") " pod="openshift-image-registry/image-registry-58dbcf67c5-dndpx"
Sep 17 17:35:18.672971 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:18.672955    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1e86520f-fe34-47cb-a4fc-2c01d1f9c9be-bound-sa-token\") pod \"image-registry-58dbcf67c5-dndpx\" (UID: \"1e86520f-fe34-47cb-a4fc-2c01d1f9c9be\") " pod="openshift-image-registry/image-registry-58dbcf67c5-dndpx"
Sep 17 17:35:18.740175 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:18.740115    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j4ds_d3f553e8-b416-4d7d-bbef-991a0404b287/konnectivity-agent/0.log" limit=1024
Sep 17 17:35:18.786217 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:18.786192    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-58dbcf67c5-dndpx"
Sep 17 17:35:18.834645 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:18.834616    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j4ds_d3f553e8-b416-4d7d-bbef-991a0404b287/konnectivity-agent/0.log" limit=1024
Sep 17 17:35:18.958648 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:18.958607    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j4ds_d3f553e8-b416-4d7d-bbef-991a0404b287/konnectivity-agent/0.log" limit=1024
Sep 17 17:35:19.082406 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:19.082316    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-58dbcf67c5-dndpx"]
Sep 17 17:35:19.086198 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:35:19.086169    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e86520f_fe34_47cb_a4fc_2c01d1f9c9be.slice/crio-7a9583f4452c8a9ca72170abe3f0ac8ab97fe9520f44617f5fc76a4d4ad6bd76 WatchSource:0}: Error finding container 7a9583f4452c8a9ca72170abe3f0ac8ab97fe9520f44617f5fc76a4d4ad6bd76: Status 404 returned error can't find the container with id 7a9583f4452c8a9ca72170abe3f0ac8ab97fe9520f44617f5fc76a4d4ad6bd76
Sep 17 17:35:19.416505 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:19.416412    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-58dbcf67c5-dndpx" event={"ID":"1e86520f-fe34-47cb-a4fc-2c01d1f9c9be","Type":"ContainerStarted","Data":"a2248e6a2fc9296445d875e94531d6d47115dd602e8c10350bcdf855fc0afdd9"}
Sep 17 17:35:19.416505 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:19.416473    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-58dbcf67c5-dndpx" event={"ID":"1e86520f-fe34-47cb-a4fc-2c01d1f9c9be","Type":"ContainerStarted","Data":"7a9583f4452c8a9ca72170abe3f0ac8ab97fe9520f44617f5fc76a4d4ad6bd76"}
Sep 17 17:35:19.416701 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:19.416572    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-58dbcf67c5-dndpx"
Sep 17 17:35:19.440381 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:19.440317    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-58dbcf67c5-dndpx" podStartSLOduration=1.440296349 podStartE2EDuration="1.440296349s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:35:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:35:19.438677553 +0000 UTC m=+74.627551268" watchObservedRunningTime="2026-09-17 17:35:19.440296349 +0000 UTC m=+74.629170058"
Sep 17 17:35:20.057574 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:20.057537    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j4ds_d3f553e8-b416-4d7d-bbef-991a0404b287/konnectivity-agent/0.log" limit=1024
Sep 17 17:35:20.163034 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:20.163002    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j4ds_d3f553e8-b416-4d7d-bbef-991a0404b287/konnectivity-agent/0.log" limit=1024
Sep 17 17:35:20.312684 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:20.312605    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j4ds_d3f553e8-b416-4d7d-bbef-991a0404b287/konnectivity-agent/0.log" limit=1024
Sep 17 17:35:20.421259 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:20.421224    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-5ct2x" event={"ID":"b3d9c617-b694-40c2-98ce-3772a892df3c","Type":"ContainerStarted","Data":"4482b7e8b6e553b19f5526d3223a8018ee1f639fa293533e399109546910a754"}
Sep 17 17:35:20.443757 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:20.443707    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-5ct2x" podStartSLOduration=33.958613646 podStartE2EDuration="35.443692304s" totalImagesPullingTime="1.485078658s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:34:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:35:20.442862431 +0000 UTC m=+75.631736137" watchObservedRunningTime="2026-09-17 17:35:20.443692304 +0000 UTC m=+75.632566009"
Sep 17 17:35:20.685063 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:20.684967    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j4ds_d3f553e8-b416-4d7d-bbef-991a0404b287/konnectivity-agent/0.log" limit=1024
Sep 17 17:35:20.772756 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:20.772721    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j4ds_d3f553e8-b416-4d7d-bbef-991a0404b287/konnectivity-agent/0.log" limit=1024
Sep 17 17:35:20.846454 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:20.846426    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j4ds_d3f553e8-b416-4d7d-bbef-991a0404b287/konnectivity-agent/0.log" limit=1024
Sep 17 17:35:24.341178 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:24.341147    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j4ds_d3f553e8-b416-4d7d-bbef-991a0404b287/konnectivity-agent/0.log" limit=1024
Sep 17 17:35:24.400181 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:24.400158    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-tnkgs"
Sep 17 17:35:24.422259 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:24.422232    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j4ds_d3f553e8-b416-4d7d-bbef-991a0404b287/konnectivity-agent/0.log" limit=1024
Sep 17 17:35:24.697754 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:24.697677    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/76b84a3e-8b66-432d-b9f3-4aeaf96d262f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kpnbq\" (UID: \"76b84a3e-8b66-432d-b9f3-4aeaf96d262f\") " pod="openshift-insights/insights-runtime-extractor-kpnbq"
Sep 17 17:35:24.699911 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:24.699883    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/76b84a3e-8b66-432d-b9f3-4aeaf96d262f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kpnbq\" (UID: \"76b84a3e-8b66-432d-b9f3-4aeaf96d262f\") " pod="openshift-insights/insights-runtime-extractor-kpnbq"
Sep 17 17:35:24.922950 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:24.922920    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j4ds_d3f553e8-b416-4d7d-bbef-991a0404b287/konnectivity-agent/0.log" limit=1024
Sep 17 17:35:24.945326 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:24.945303    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-7fflr\""
Sep 17 17:35:24.953774 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:24.953716    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-kpnbq"
Sep 17 17:35:25.252898 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:25.252874    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-kpnbq"]
Sep 17 17:35:25.440340 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:25.440310    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-kpnbq" event={"ID":"76b84a3e-8b66-432d-b9f3-4aeaf96d262f","Type":"ContainerStarted","Data":"069f331647b726aa70d645d956176d807104d42266b3ad4ef41e7cca58014230"}
Sep 17 17:35:25.440644 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:25.440373    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-kpnbq" event={"ID":"76b84a3e-8b66-432d-b9f3-4aeaf96d262f","Type":"ContainerStarted","Data":"13ed78cfc6451dd2c131f09972b53ce71ed535a096781a773232fbc5952ded9a"}
Sep 17 17:35:26.198365 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:26.198326    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j4ds_d3f553e8-b416-4d7d-bbef-991a0404b287/konnectivity-agent/0.log" limit=1024
Sep 17 17:35:26.445755 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:26.445724    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-kpnbq" event={"ID":"76b84a3e-8b66-432d-b9f3-4aeaf96d262f","Type":"ContainerStarted","Data":"bc2e74d3d8c3b231244606b07456b8c08b804c9898cd517a821b6007213facf2"}
Sep 17 17:35:27.266369 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:27.266325    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j4ds_d3f553e8-b416-4d7d-bbef-991a0404b287/konnectivity-agent/0.log" limit=1024
Sep 17 17:35:27.653019 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:27.652954    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-dd5c969b-rmtb5"]
Sep 17 17:35:27.674531 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:27.674507    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-dd5c969b-rmtb5"]
Sep 17 17:35:27.674667 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:27.674595    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-dd5c969b-rmtb5"
Sep 17 17:35:27.677618 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:27.677596    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 17 17:35:27.677733 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:27.677624    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 17 17:35:27.677733 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:27.677600    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 17 17:35:27.677733 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:27.677596    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 17 17:35:27.677733 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:27.677683    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 17 17:35:27.678576 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:27.678559    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 17 17:35:27.678576 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:27.678568    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-vkgxj\""
Sep 17 17:35:27.678718 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:27.678656    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 17 17:35:27.720147 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:27.720120    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/035b396a-6ac7-4fe0-a639-104d2ff1bad3-console-oauth-config\") pod \"console-dd5c969b-rmtb5\" (UID: \"035b396a-6ac7-4fe0-a639-104d2ff1bad3\") " pod="openshift-console/console-dd5c969b-rmtb5"
Sep 17 17:35:27.720260 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:27.720165    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-trl25\" (UniqueName: \"kubernetes.io/projected/035b396a-6ac7-4fe0-a639-104d2ff1bad3-kube-api-access-trl25\") pod \"console-dd5c969b-rmtb5\" (UID: \"035b396a-6ac7-4fe0-a639-104d2ff1bad3\") " pod="openshift-console/console-dd5c969b-rmtb5"
Sep 17 17:35:27.720260 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:27.720187    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/035b396a-6ac7-4fe0-a639-104d2ff1bad3-oauth-serving-cert\") pod \"console-dd5c969b-rmtb5\" (UID: \"035b396a-6ac7-4fe0-a639-104d2ff1bad3\") " pod="openshift-console/console-dd5c969b-rmtb5"
Sep 17 17:35:27.720260 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:27.720206    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/035b396a-6ac7-4fe0-a639-104d2ff1bad3-console-serving-cert\") pod \"console-dd5c969b-rmtb5\" (UID: \"035b396a-6ac7-4fe0-a639-104d2ff1bad3\") " pod="openshift-console/console-dd5c969b-rmtb5"
Sep 17 17:35:27.720260 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:27.720222    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/035b396a-6ac7-4fe0-a639-104d2ff1bad3-console-config\") pod \"console-dd5c969b-rmtb5\" (UID: \"035b396a-6ac7-4fe0-a639-104d2ff1bad3\") " pod="openshift-console/console-dd5c969b-rmtb5"
Sep 17 17:35:27.720260 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:27.720240    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/035b396a-6ac7-4fe0-a639-104d2ff1bad3-service-ca\") pod \"console-dd5c969b-rmtb5\" (UID: \"035b396a-6ac7-4fe0-a639-104d2ff1bad3\") " pod="openshift-console/console-dd5c969b-rmtb5"
Sep 17 17:35:27.820711 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:27.820680    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/035b396a-6ac7-4fe0-a639-104d2ff1bad3-oauth-serving-cert\") pod \"console-dd5c969b-rmtb5\" (UID: \"035b396a-6ac7-4fe0-a639-104d2ff1bad3\") " pod="openshift-console/console-dd5c969b-rmtb5"
Sep 17 17:35:27.820839 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:27.820721    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/035b396a-6ac7-4fe0-a639-104d2ff1bad3-console-serving-cert\") pod \"console-dd5c969b-rmtb5\" (UID: \"035b396a-6ac7-4fe0-a639-104d2ff1bad3\") " pod="openshift-console/console-dd5c969b-rmtb5"
Sep 17 17:35:27.820839 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:27.820745    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/035b396a-6ac7-4fe0-a639-104d2ff1bad3-console-config\") pod \"console-dd5c969b-rmtb5\" (UID: \"035b396a-6ac7-4fe0-a639-104d2ff1bad3\") " pod="openshift-console/console-dd5c969b-rmtb5"
Sep 17 17:35:27.820839 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:27.820775    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/035b396a-6ac7-4fe0-a639-104d2ff1bad3-service-ca\") pod \"console-dd5c969b-rmtb5\" (UID: \"035b396a-6ac7-4fe0-a639-104d2ff1bad3\") " pod="openshift-console/console-dd5c969b-rmtb5"
Sep 17 17:35:27.820839 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:27.820812    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/035b396a-6ac7-4fe0-a639-104d2ff1bad3-console-oauth-config\") pod \"console-dd5c969b-rmtb5\" (UID: \"035b396a-6ac7-4fe0-a639-104d2ff1bad3\") " pod="openshift-console/console-dd5c969b-rmtb5"
Sep 17 17:35:27.821030 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:27.820859    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-trl25\" (UniqueName: \"kubernetes.io/projected/035b396a-6ac7-4fe0-a639-104d2ff1bad3-kube-api-access-trl25\") pod \"console-dd5c969b-rmtb5\" (UID: \"035b396a-6ac7-4fe0-a639-104d2ff1bad3\") " pod="openshift-console/console-dd5c969b-rmtb5"
Sep 17 17:35:27.821493 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:27.821468    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/035b396a-6ac7-4fe0-a639-104d2ff1bad3-oauth-serving-cert\") pod \"console-dd5c969b-rmtb5\" (UID: \"035b396a-6ac7-4fe0-a639-104d2ff1bad3\") " pod="openshift-console/console-dd5c969b-rmtb5"
Sep 17 17:35:27.821594 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:27.821501    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/035b396a-6ac7-4fe0-a639-104d2ff1bad3-service-ca\") pod \"console-dd5c969b-rmtb5\" (UID: \"035b396a-6ac7-4fe0-a639-104d2ff1bad3\") " pod="openshift-console/console-dd5c969b-rmtb5"
Sep 17 17:35:27.821594 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:27.821527    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/035b396a-6ac7-4fe0-a639-104d2ff1bad3-console-config\") pod \"console-dd5c969b-rmtb5\" (UID: \"035b396a-6ac7-4fe0-a639-104d2ff1bad3\") " pod="openshift-console/console-dd5c969b-rmtb5"
Sep 17 17:35:27.823230 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:27.823211    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/035b396a-6ac7-4fe0-a639-104d2ff1bad3-console-serving-cert\") pod \"console-dd5c969b-rmtb5\" (UID: \"035b396a-6ac7-4fe0-a639-104d2ff1bad3\") " pod="openshift-console/console-dd5c969b-rmtb5"
Sep 17 17:35:27.823487 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:27.823468    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/035b396a-6ac7-4fe0-a639-104d2ff1bad3-console-oauth-config\") pod \"console-dd5c969b-rmtb5\" (UID: \"035b396a-6ac7-4fe0-a639-104d2ff1bad3\") " pod="openshift-console/console-dd5c969b-rmtb5"
Sep 17 17:35:27.830763 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:27.830745    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-trl25\" (UniqueName: \"kubernetes.io/projected/035b396a-6ac7-4fe0-a639-104d2ff1bad3-kube-api-access-trl25\") pod \"console-dd5c969b-rmtb5\" (UID: \"035b396a-6ac7-4fe0-a639-104d2ff1bad3\") " pod="openshift-console/console-dd5c969b-rmtb5"
Sep 17 17:35:27.983784 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:27.983715    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-dd5c969b-rmtb5"
Sep 17 17:35:28.324902 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:28.324877    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j4ds_d3f553e8-b416-4d7d-bbef-991a0404b287/konnectivity-agent/0.log" limit=1024
Sep 17 17:35:28.357399 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:28.357374    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-dd5c969b-rmtb5"]
Sep 17 17:35:28.360883 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:35:28.360861    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod035b396a_6ac7_4fe0_a639_104d2ff1bad3.slice/crio-87d379760ebda8e3c8eb10796b2913d359334ebd3ff43da95547299e20f74b45 WatchSource:0}: Error finding container 87d379760ebda8e3c8eb10796b2913d359334ebd3ff43da95547299e20f74b45: Status 404 returned error can't find the container with id 87d379760ebda8e3c8eb10796b2913d359334ebd3ff43da95547299e20f74b45
Sep 17 17:35:28.424961 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:28.424936    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j4ds_d3f553e8-b416-4d7d-bbef-991a0404b287/konnectivity-agent/0.log" limit=1024
Sep 17 17:35:28.453603 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:28.453579    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-dd5c969b-rmtb5" event={"ID":"035b396a-6ac7-4fe0-a639-104d2ff1bad3","Type":"ContainerStarted","Data":"87d379760ebda8e3c8eb10796b2913d359334ebd3ff43da95547299e20f74b45"}
Sep 17 17:35:28.455376 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:28.455354    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-kpnbq" event={"ID":"76b84a3e-8b66-432d-b9f3-4aeaf96d262f","Type":"ContainerStarted","Data":"862227f5be7db2d199735b0d63f285d40d565fbb8fffd72496884aa88acfbdfb"}
Sep 17 17:35:28.477174 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:28.477108    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-kpnbq" podStartSLOduration=33.85978026 podStartE2EDuration="36.477095968s" totalImagesPullingTime="2.617315708s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:34:52 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:35:28.476728481 +0000 UTC m=+83.665602183" watchObservedRunningTime="2026-09-17 17:35:28.477095968 +0000 UTC m=+83.665969674"
Sep 17 17:35:29.210770 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:29.210706    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j4ds_d3f553e8-b416-4d7d-bbef-991a0404b287/konnectivity-agent/0.log" limit=1024
Sep 17 17:35:29.349646 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:29.349571    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j4ds_d3f553e8-b416-4d7d-bbef-991a0404b287/konnectivity-agent/0.log" limit=1024
Sep 17 17:35:29.501756 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:29.501722    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j4ds_d3f553e8-b416-4d7d-bbef-991a0404b287/konnectivity-agent/0.log" limit=1024
Sep 17 17:35:29.697074 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:29.697024    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j4ds_d3f553e8-b416-4d7d-bbef-991a0404b287/konnectivity-agent/0.log" limit=1024
Sep 17 17:35:29.791560 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:29.791472    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j4ds_d3f553e8-b416-4d7d-bbef-991a0404b287/konnectivity-agent/0.log" limit=1024
Sep 17 17:35:29.836064 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:29.836016    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-84c885658c-tzmkz"]
Sep 17 17:35:29.842740 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:29.842715    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-84c885658c-tzmkz"
Sep 17 17:35:29.845530 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:29.845508    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Sep 17 17:35:29.846733 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:29.846709    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Sep 17 17:35:29.847714 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:29.847684    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 17 17:35:29.848401 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:29.848369    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-j84lq\""
Sep 17 17:35:29.850202 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:29.850151    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Sep 17 17:35:29.859875 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:29.859854    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-84c885658c-tzmkz"]
Sep 17 17:35:29.876845 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:29.876826    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-x5649"]
Sep 17 17:35:29.879394 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:29.879377    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-x5649"
Sep 17 17:35:29.881777 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:29.881752    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 17 17:35:29.881892 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:29.881873    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 17 17:35:29.882034 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:29.882018    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-m5jv7\""
Sep 17 17:35:29.882247 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:29.882162    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 17 17:35:29.916149 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:29.916117    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j4ds_d3f553e8-b416-4d7d-bbef-991a0404b287/konnectivity-agent/0.log" limit=1024
Sep 17 17:35:29.940238 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:29.940203    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1d665736-4c4e-49aa-ab27-8ab235e46063-metrics-client-ca\") pod \"kube-state-metrics-84c885658c-tzmkz\" (UID: \"1d665736-4c4e-49aa-ab27-8ab235e46063\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-tzmkz"
Sep 17 17:35:29.940386 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:29.940252    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/70b48508-2726-4415-ba31-35246e0f6ad1-metrics-client-ca\") pod \"node-exporter-x5649\" (UID: \"70b48508-2726-4415-ba31-35246e0f6ad1\") " pod="openshift-monitoring/node-exporter-x5649"
Sep 17 17:35:29.940386 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:29.940282    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1d665736-4c4e-49aa-ab27-8ab235e46063-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-84c885658c-tzmkz\" (UID: \"1d665736-4c4e-49aa-ab27-8ab235e46063\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-tzmkz"
Sep 17 17:35:29.940386 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:29.940309    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vlb8g\" (UniqueName: \"kubernetes.io/projected/1d665736-4c4e-49aa-ab27-8ab235e46063-kube-api-access-vlb8g\") pod \"kube-state-metrics-84c885658c-tzmkz\" (UID: \"1d665736-4c4e-49aa-ab27-8ab235e46063\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-tzmkz"
Sep 17 17:35:29.940386 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:29.940340    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/70b48508-2726-4415-ba31-35246e0f6ad1-node-exporter-accelerators-collector-config\") pod \"node-exporter-x5649\" (UID: \"70b48508-2726-4415-ba31-35246e0f6ad1\") " pod="openshift-monitoring/node-exporter-x5649"
Sep 17 17:35:29.940600 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:29.940470    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cnlb5\" (UniqueName: \"kubernetes.io/projected/70b48508-2726-4415-ba31-35246e0f6ad1-kube-api-access-cnlb5\") pod \"node-exporter-x5649\" (UID: \"70b48508-2726-4415-ba31-35246e0f6ad1\") " pod="openshift-monitoring/node-exporter-x5649"
Sep 17 17:35:29.940600 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:29.940511    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/1d665736-4c4e-49aa-ab27-8ab235e46063-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-84c885658c-tzmkz\" (UID: \"1d665736-4c4e-49aa-ab27-8ab235e46063\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-tzmkz"
Sep 17 17:35:29.940600 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:29.940535    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/70b48508-2726-4415-ba31-35246e0f6ad1-sys\") pod \"node-exporter-x5649\" (UID: \"70b48508-2726-4415-ba31-35246e0f6ad1\") " pod="openshift-monitoring/node-exporter-x5649"
Sep 17 17:35:29.940600 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:29.940581    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/70b48508-2726-4415-ba31-35246e0f6ad1-node-exporter-tls\") pod \"node-exporter-x5649\" (UID: \"70b48508-2726-4415-ba31-35246e0f6ad1\") " pod="openshift-monitoring/node-exporter-x5649"
Sep 17 17:35:29.940793 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:29.940619    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/1d665736-4c4e-49aa-ab27-8ab235e46063-kube-state-metrics-tls\") pod \"kube-state-metrics-84c885658c-tzmkz\" (UID: \"1d665736-4c4e-49aa-ab27-8ab235e46063\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-tzmkz"
Sep 17 17:35:29.940793 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:29.940644    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/70b48508-2726-4415-ba31-35246e0f6ad1-root\") pod \"node-exporter-x5649\" (UID: \"70b48508-2726-4415-ba31-35246e0f6ad1\") " pod="openshift-monitoring/node-exporter-x5649"
Sep 17 17:35:29.940793 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:29.940671    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/1d665736-4c4e-49aa-ab27-8ab235e46063-volume-directive-shadow\") pod \"kube-state-metrics-84c885658c-tzmkz\" (UID: \"1d665736-4c4e-49aa-ab27-8ab235e46063\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-tzmkz"
Sep 17 17:35:29.940793 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:29.940699    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/70b48508-2726-4415-ba31-35246e0f6ad1-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-x5649\" (UID: \"70b48508-2726-4415-ba31-35246e0f6ad1\") " pod="openshift-monitoring/node-exporter-x5649"
Sep 17 17:35:29.940793 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:29.940760    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/70b48508-2726-4415-ba31-35246e0f6ad1-node-exporter-textfile\") pod \"node-exporter-x5649\" (UID: \"70b48508-2726-4415-ba31-35246e0f6ad1\") " pod="openshift-monitoring/node-exporter-x5649"
Sep 17 17:35:29.941021 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:29.940797    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/70b48508-2726-4415-ba31-35246e0f6ad1-node-exporter-wtmp\") pod \"node-exporter-x5649\" (UID: \"70b48508-2726-4415-ba31-35246e0f6ad1\") " pod="openshift-monitoring/node-exporter-x5649"
Sep 17 17:35:30.003801 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:30.003751    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j4ds_d3f553e8-b416-4d7d-bbef-991a0404b287/konnectivity-agent/0.log" limit=1024
Sep 17 17:35:30.041835 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:30.041751    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1d665736-4c4e-49aa-ab27-8ab235e46063-metrics-client-ca\") pod \"kube-state-metrics-84c885658c-tzmkz\" (UID: \"1d665736-4c4e-49aa-ab27-8ab235e46063\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-tzmkz"
Sep 17 17:35:30.041835 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:30.041798    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/70b48508-2726-4415-ba31-35246e0f6ad1-metrics-client-ca\") pod \"node-exporter-x5649\" (UID: \"70b48508-2726-4415-ba31-35246e0f6ad1\") " pod="openshift-monitoring/node-exporter-x5649"
Sep 17 17:35:30.041835 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:30.041826    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1d665736-4c4e-49aa-ab27-8ab235e46063-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-84c885658c-tzmkz\" (UID: \"1d665736-4c4e-49aa-ab27-8ab235e46063\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-tzmkz"
Sep 17 17:35:30.042100 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:30.041849    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vlb8g\" (UniqueName: \"kubernetes.io/projected/1d665736-4c4e-49aa-ab27-8ab235e46063-kube-api-access-vlb8g\") pod \"kube-state-metrics-84c885658c-tzmkz\" (UID: \"1d665736-4c4e-49aa-ab27-8ab235e46063\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-tzmkz"
Sep 17 17:35:30.042100 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:30.041875    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/70b48508-2726-4415-ba31-35246e0f6ad1-node-exporter-accelerators-collector-config\") pod \"node-exporter-x5649\" (UID: \"70b48508-2726-4415-ba31-35246e0f6ad1\") " pod="openshift-monitoring/node-exporter-x5649"
Sep 17 17:35:30.042100 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:30.041904    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cnlb5\" (UniqueName: \"kubernetes.io/projected/70b48508-2726-4415-ba31-35246e0f6ad1-kube-api-access-cnlb5\") pod \"node-exporter-x5649\" (UID: \"70b48508-2726-4415-ba31-35246e0f6ad1\") " pod="openshift-monitoring/node-exporter-x5649"
Sep 17 17:35:30.042100 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:30.041934    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/1d665736-4c4e-49aa-ab27-8ab235e46063-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-84c885658c-tzmkz\" (UID: \"1d665736-4c4e-49aa-ab27-8ab235e46063\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-tzmkz"
Sep 17 17:35:30.042100 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:30.041956    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/70b48508-2726-4415-ba31-35246e0f6ad1-sys\") pod \"node-exporter-x5649\" (UID: \"70b48508-2726-4415-ba31-35246e0f6ad1\") " pod="openshift-monitoring/node-exporter-x5649"
Sep 17 17:35:30.042100 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:30.041982    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/70b48508-2726-4415-ba31-35246e0f6ad1-node-exporter-tls\") pod \"node-exporter-x5649\" (UID: \"70b48508-2726-4415-ba31-35246e0f6ad1\") " pod="openshift-monitoring/node-exporter-x5649"
Sep 17 17:35:30.042100 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:30.042024    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/1d665736-4c4e-49aa-ab27-8ab235e46063-kube-state-metrics-tls\") pod \"kube-state-metrics-84c885658c-tzmkz\" (UID: \"1d665736-4c4e-49aa-ab27-8ab235e46063\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-tzmkz"
Sep 17 17:35:30.042100 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:30.042073    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/70b48508-2726-4415-ba31-35246e0f6ad1-root\") pod \"node-exporter-x5649\" (UID: \"70b48508-2726-4415-ba31-35246e0f6ad1\") " pod="openshift-monitoring/node-exporter-x5649"
Sep 17 17:35:30.042447 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:30.042106    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/1d665736-4c4e-49aa-ab27-8ab235e46063-volume-directive-shadow\") pod \"kube-state-metrics-84c885658c-tzmkz\" (UID: \"1d665736-4c4e-49aa-ab27-8ab235e46063\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-tzmkz"
Sep 17 17:35:30.042447 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:30.042138    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/70b48508-2726-4415-ba31-35246e0f6ad1-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-x5649\" (UID: \"70b48508-2726-4415-ba31-35246e0f6ad1\") " pod="openshift-monitoring/node-exporter-x5649"
Sep 17 17:35:30.042447 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:30.042177    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/70b48508-2726-4415-ba31-35246e0f6ad1-node-exporter-textfile\") pod \"node-exporter-x5649\" (UID: \"70b48508-2726-4415-ba31-35246e0f6ad1\") " pod="openshift-monitoring/node-exporter-x5649"
Sep 17 17:35:30.042447 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:30.042201    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/70b48508-2726-4415-ba31-35246e0f6ad1-node-exporter-wtmp\") pod \"node-exporter-x5649\" (UID: \"70b48508-2726-4415-ba31-35246e0f6ad1\") " pod="openshift-monitoring/node-exporter-x5649"
Sep 17 17:35:30.042447 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:30.042379    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/70b48508-2726-4415-ba31-35246e0f6ad1-node-exporter-wtmp\") pod \"node-exporter-x5649\" (UID: \"70b48508-2726-4415-ba31-35246e0f6ad1\") " pod="openshift-monitoring/node-exporter-x5649"
Sep 17 17:35:30.043471 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:30.042934    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/70b48508-2726-4415-ba31-35246e0f6ad1-root\") pod \"node-exporter-x5649\" (UID: \"70b48508-2726-4415-ba31-35246e0f6ad1\") " pod="openshift-monitoring/node-exporter-x5649"
Sep 17 17:35:30.043471 ip-10-0-134-96 kubenswrapper[2574]: E0917 17:35:30.043031    2574 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Sep 17 17:35:30.043471 ip-10-0-134-96 kubenswrapper[2574]: E0917 17:35:30.043107    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/70b48508-2726-4415-ba31-35246e0f6ad1-node-exporter-tls podName:70b48508-2726-4415-ba31-35246e0f6ad1 nodeName:}" failed. No retries permitted until 2026-09-17 17:35:30.543088334 +0000 UTC m=+85.731962021 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/70b48508-2726-4415-ba31-35246e0f6ad1-node-exporter-tls") pod "node-exporter-x5649" (UID: "70b48508-2726-4415-ba31-35246e0f6ad1") : secret "node-exporter-tls" not found
Sep 17 17:35:30.043471 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:30.043089    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1d665736-4c4e-49aa-ab27-8ab235e46063-metrics-client-ca\") pod \"kube-state-metrics-84c885658c-tzmkz\" (UID: \"1d665736-4c4e-49aa-ab27-8ab235e46063\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-tzmkz"
Sep 17 17:35:30.043471 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:30.043111    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/70b48508-2726-4415-ba31-35246e0f6ad1-sys\") pod \"node-exporter-x5649\" (UID: \"70b48508-2726-4415-ba31-35246e0f6ad1\") " pod="openshift-monitoring/node-exporter-x5649"
Sep 17 17:35:30.043471 ip-10-0-134-96 kubenswrapper[2574]: E0917 17:35:30.043194    2574 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found
Sep 17 17:35:30.043471 ip-10-0-134-96 kubenswrapper[2574]: E0917 17:35:30.043230    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1d665736-4c4e-49aa-ab27-8ab235e46063-kube-state-metrics-tls podName:1d665736-4c4e-49aa-ab27-8ab235e46063 nodeName:}" failed. No retries permitted until 2026-09-17 17:35:30.543217974 +0000 UTC m=+85.732091659 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/1d665736-4c4e-49aa-ab27-8ab235e46063-kube-state-metrics-tls") pod "kube-state-metrics-84c885658c-tzmkz" (UID: "1d665736-4c4e-49aa-ab27-8ab235e46063") : secret "kube-state-metrics-tls" not found
Sep 17 17:35:30.043471 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:30.043331    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/1d665736-4c4e-49aa-ab27-8ab235e46063-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-84c885658c-tzmkz\" (UID: \"1d665736-4c4e-49aa-ab27-8ab235e46063\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-tzmkz"
Sep 17 17:35:30.043471 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:30.043374    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/1d665736-4c4e-49aa-ab27-8ab235e46063-volume-directive-shadow\") pod \"kube-state-metrics-84c885658c-tzmkz\" (UID: \"1d665736-4c4e-49aa-ab27-8ab235e46063\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-tzmkz"
Sep 17 17:35:30.043471 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:30.043437    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/70b48508-2726-4415-ba31-35246e0f6ad1-metrics-client-ca\") pod \"node-exporter-x5649\" (UID: \"70b48508-2726-4415-ba31-35246e0f6ad1\") " pod="openshift-monitoring/node-exporter-x5649"
Sep 17 17:35:30.044061 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:30.043606    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/70b48508-2726-4415-ba31-35246e0f6ad1-node-exporter-textfile\") pod \"node-exporter-x5649\" (UID: \"70b48508-2726-4415-ba31-35246e0f6ad1\") " pod="openshift-monitoring/node-exporter-x5649"
Sep 17 17:35:30.044061 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:30.043619    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/70b48508-2726-4415-ba31-35246e0f6ad1-node-exporter-accelerators-collector-config\") pod \"node-exporter-x5649\" (UID: \"70b48508-2726-4415-ba31-35246e0f6ad1\") " pod="openshift-monitoring/node-exporter-x5649"
Sep 17 17:35:30.045922 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:30.045894    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/70b48508-2726-4415-ba31-35246e0f6ad1-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-x5649\" (UID: \"70b48508-2726-4415-ba31-35246e0f6ad1\") " pod="openshift-monitoring/node-exporter-x5649"
Sep 17 17:35:30.046153 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:30.046108    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1d665736-4c4e-49aa-ab27-8ab235e46063-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-84c885658c-tzmkz\" (UID: \"1d665736-4c4e-49aa-ab27-8ab235e46063\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-tzmkz"
Sep 17 17:35:30.056434 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:30.056410    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vlb8g\" (UniqueName: \"kubernetes.io/projected/1d665736-4c4e-49aa-ab27-8ab235e46063-kube-api-access-vlb8g\") pod \"kube-state-metrics-84c885658c-tzmkz\" (UID: \"1d665736-4c4e-49aa-ab27-8ab235e46063\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-tzmkz"
Sep 17 17:35:30.057123 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:30.057079    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cnlb5\" (UniqueName: \"kubernetes.io/projected/70b48508-2726-4415-ba31-35246e0f6ad1-kube-api-access-cnlb5\") pod \"node-exporter-x5649\" (UID: \"70b48508-2726-4415-ba31-35246e0f6ad1\") " pod="openshift-monitoring/node-exporter-x5649"
Sep 17 17:35:30.157821 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:30.157725    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j4ds_d3f553e8-b416-4d7d-bbef-991a0404b287/konnectivity-agent/0.log" limit=1024
Sep 17 17:35:30.253157 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:30.253114    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j4ds_d3f553e8-b416-4d7d-bbef-991a0404b287/konnectivity-agent/0.log" limit=1024
Sep 17 17:35:30.353953 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:30.353886    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j4ds_d3f553e8-b416-4d7d-bbef-991a0404b287/konnectivity-agent/0.log" limit=1024
Sep 17 17:35:30.452349 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:30.452305    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j4ds_d3f553e8-b416-4d7d-bbef-991a0404b287/konnectivity-agent/0.log" limit=1024
Sep 17 17:35:30.547024 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:30.546991    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/70b48508-2726-4415-ba31-35246e0f6ad1-node-exporter-tls\") pod \"node-exporter-x5649\" (UID: \"70b48508-2726-4415-ba31-35246e0f6ad1\") " pod="openshift-monitoring/node-exporter-x5649"
Sep 17 17:35:30.547213 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:30.547055    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/1d665736-4c4e-49aa-ab27-8ab235e46063-kube-state-metrics-tls\") pod \"kube-state-metrics-84c885658c-tzmkz\" (UID: \"1d665736-4c4e-49aa-ab27-8ab235e46063\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-tzmkz"
Sep 17 17:35:30.549611 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:30.549584    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/70b48508-2726-4415-ba31-35246e0f6ad1-node-exporter-tls\") pod \"node-exporter-x5649\" (UID: \"70b48508-2726-4415-ba31-35246e0f6ad1\") " pod="openshift-monitoring/node-exporter-x5649"
Sep 17 17:35:30.549728 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:30.549705    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/1d665736-4c4e-49aa-ab27-8ab235e46063-kube-state-metrics-tls\") pod \"kube-state-metrics-84c885658c-tzmkz\" (UID: \"1d665736-4c4e-49aa-ab27-8ab235e46063\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-tzmkz"
Sep 17 17:35:30.755748 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:30.755714    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-84c885658c-tzmkz"
Sep 17 17:35:30.791753 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:30.791722    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-x5649"
Sep 17 17:35:30.813238 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:30.813211    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j4ds_d3f553e8-b416-4d7d-bbef-991a0404b287/konnectivity-agent/0.log" limit=1024
Sep 17 17:35:30.929803 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:30.929770    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j4ds_d3f553e8-b416-4d7d-bbef-991a0404b287/konnectivity-agent/0.log" limit=1024
Sep 17 17:35:31.083500 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:31.083422    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j4ds_d3f553e8-b416-4d7d-bbef-991a0404b287/konnectivity-agent/0.log" limit=1024
Sep 17 17:35:32.039790 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:35:32.039623    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70b48508_2726_4415_ba31_35246e0f6ad1.slice/crio-59cb0ff2f8102d95afd0aae0f9d8260c6688bace930be80339264cbec016062e WatchSource:0}: Error finding container 59cb0ff2f8102d95afd0aae0f9d8260c6688bace930be80339264cbec016062e: Status 404 returned error can't find the container with id 59cb0ff2f8102d95afd0aae0f9d8260c6688bace930be80339264cbec016062e
Sep 17 17:35:32.304081 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:32.303994    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-84c885658c-tzmkz"]
Sep 17 17:35:32.307128 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:35:32.307099    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d665736_4c4e_49aa_ab27_8ab235e46063.slice/crio-fb616d8532e0e0e0a7209817f2c76d8f38f62617e5cb2dec9e56c6dd23fbfa2a WatchSource:0}: Error finding container fb616d8532e0e0e0a7209817f2c76d8f38f62617e5cb2dec9e56c6dd23fbfa2a: Status 404 returned error can't find the container with id fb616d8532e0e0e0a7209817f2c76d8f38f62617e5cb2dec9e56c6dd23fbfa2a
Sep 17 17:35:32.388304 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:32.388280    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j4ds_d3f553e8-b416-4d7d-bbef-991a0404b287/konnectivity-agent/0.log" limit=1024
Sep 17 17:35:32.474316 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:32.474275    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-dd5c969b-rmtb5" event={"ID":"035b396a-6ac7-4fe0-a639-104d2ff1bad3","Type":"ContainerStarted","Data":"268e15ec302482efe760f1076432389c849834d3885279743f08141e998eb8ed"}
Sep 17 17:35:32.475650 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:32.475613    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-84c885658c-tzmkz" event={"ID":"1d665736-4c4e-49aa-ab27-8ab235e46063","Type":"ContainerStarted","Data":"fb616d8532e0e0e0a7209817f2c76d8f38f62617e5cb2dec9e56c6dd23fbfa2a"}
Sep 17 17:35:32.477125 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:32.477092    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-x5649" event={"ID":"70b48508-2726-4415-ba31-35246e0f6ad1","Type":"ContainerStarted","Data":"59cb0ff2f8102d95afd0aae0f9d8260c6688bace930be80339264cbec016062e"}
Sep 17 17:35:32.487149 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:32.487119    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j4ds_d3f553e8-b416-4d7d-bbef-991a0404b287/konnectivity-agent/0.log" limit=1024
Sep 17 17:35:32.496431 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:32.496378    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-dd5c969b-rmtb5" podStartSLOduration=1.789959232 podStartE2EDuration="5.496363996s" totalImagesPullingTime="3.706404764s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:35:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:35:32.496356101 +0000 UTC m=+87.685229809" watchObservedRunningTime="2026-09-17 17:35:32.496363996 +0000 UTC m=+87.685237701"
Sep 17 17:35:32.755539 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:32.755507    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j4ds_d3f553e8-b416-4d7d-bbef-991a0404b287/konnectivity-agent/0.log" limit=1024
Sep 17 17:35:32.855578 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:32.855548    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j4ds_d3f553e8-b416-4d7d-bbef-991a0404b287/konnectivity-agent/0.log" limit=1024
Sep 17 17:35:33.050423 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:33.050393    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j4ds_d3f553e8-b416-4d7d-bbef-991a0404b287/konnectivity-agent/0.log" limit=1024
Sep 17 17:35:33.120683 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:33.120655    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j4ds_d3f553e8-b416-4d7d-bbef-991a0404b287/konnectivity-agent/0.log" limit=1024
Sep 17 17:35:33.485106 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:33.485077    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-84c885658c-tzmkz" event={"ID":"1d665736-4c4e-49aa-ab27-8ab235e46063","Type":"ContainerStarted","Data":"c46e3cbb95832f78dac1dfe54f6f43767f6c4fe0fdfa14f3b2611967e297e1b4"}
Sep 17 17:35:33.487364 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:33.487275    2574 generic.go:356] "Generic (PLEG): container finished" podID="70b48508-2726-4415-ba31-35246e0f6ad1" containerID="646d5389017287cccddabbb181cec09d99a9077e3acbca7f709d193b4536d3d1" exitCode=0
Sep 17 17:35:33.487364 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:33.487321    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-x5649" event={"ID":"70b48508-2726-4415-ba31-35246e0f6ad1","Type":"ContainerDied","Data":"646d5389017287cccddabbb181cec09d99a9077e3acbca7f709d193b4536d3d1"}
Sep 17 17:35:33.573312 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:33.573067    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j4ds_d3f553e8-b416-4d7d-bbef-991a0404b287/konnectivity-agent/0.log" limit=1024
Sep 17 17:35:33.805920 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:33.805887    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j4ds_d3f553e8-b416-4d7d-bbef-991a0404b287/konnectivity-agent/0.log" limit=1024
Sep 17 17:35:33.906357 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:33.906271    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j4ds_d3f553e8-b416-4d7d-bbef-991a0404b287/konnectivity-agent/0.log" limit=1024
Sep 17 17:35:34.234108 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:34.234018    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-558488469d-rfr47"]
Sep 17 17:35:34.236441 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:34.236424    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-558488469d-rfr47"
Sep 17 17:35:34.240011 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:34.239983    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Sep 17 17:35:34.240118 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:34.240034    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-4enrehj2o38jj\""
Sep 17 17:35:34.240118 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:34.240054    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Sep 17 17:35:34.240118 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:34.240098    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Sep 17 17:35:34.240118 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:34.240103    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-tdwpm\""
Sep 17 17:35:34.240118 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:34.240111    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Sep 17 17:35:34.250537 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:34.250519    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-558488469d-rfr47"]
Sep 17 17:35:34.379307 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:34.379276    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/199968f8-4c75-40d8-bc3c-2652edf4c7da-metrics-server-audit-profiles\") pod \"metrics-server-558488469d-rfr47\" (UID: \"199968f8-4c75-40d8-bc3c-2652edf4c7da\") " pod="openshift-monitoring/metrics-server-558488469d-rfr47"
Sep 17 17:35:34.379450 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:34.379313    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/199968f8-4c75-40d8-bc3c-2652edf4c7da-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-558488469d-rfr47\" (UID: \"199968f8-4c75-40d8-bc3c-2652edf4c7da\") " pod="openshift-monitoring/metrics-server-558488469d-rfr47"
Sep 17 17:35:34.379450 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:34.379334    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/199968f8-4c75-40d8-bc3c-2652edf4c7da-client-ca-bundle\") pod \"metrics-server-558488469d-rfr47\" (UID: \"199968f8-4c75-40d8-bc3c-2652edf4c7da\") " pod="openshift-monitoring/metrics-server-558488469d-rfr47"
Sep 17 17:35:34.379450 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:34.379370    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/199968f8-4c75-40d8-bc3c-2652edf4c7da-secret-metrics-server-client-certs\") pod \"metrics-server-558488469d-rfr47\" (UID: \"199968f8-4c75-40d8-bc3c-2652edf4c7da\") " pod="openshift-monitoring/metrics-server-558488469d-rfr47"
Sep 17 17:35:34.379450 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:34.379387    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pdqnm\" (UniqueName: \"kubernetes.io/projected/199968f8-4c75-40d8-bc3c-2652edf4c7da-kube-api-access-pdqnm\") pod \"metrics-server-558488469d-rfr47\" (UID: \"199968f8-4c75-40d8-bc3c-2652edf4c7da\") " pod="openshift-monitoring/metrics-server-558488469d-rfr47"
Sep 17 17:35:34.379450 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:34.379417    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/199968f8-4c75-40d8-bc3c-2652edf4c7da-secret-metrics-server-tls\") pod \"metrics-server-558488469d-rfr47\" (UID: \"199968f8-4c75-40d8-bc3c-2652edf4c7da\") " pod="openshift-monitoring/metrics-server-558488469d-rfr47"
Sep 17 17:35:34.379450 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:34.379447    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/199968f8-4c75-40d8-bc3c-2652edf4c7da-audit-log\") pod \"metrics-server-558488469d-rfr47\" (UID: \"199968f8-4c75-40d8-bc3c-2652edf4c7da\") " pod="openshift-monitoring/metrics-server-558488469d-rfr47"
Sep 17 17:35:34.480713 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:34.480683    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/199968f8-4c75-40d8-bc3c-2652edf4c7da-secret-metrics-server-tls\") pod \"metrics-server-558488469d-rfr47\" (UID: \"199968f8-4c75-40d8-bc3c-2652edf4c7da\") " pod="openshift-monitoring/metrics-server-558488469d-rfr47"
Sep 17 17:35:34.480841 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:34.480718    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/199968f8-4c75-40d8-bc3c-2652edf4c7da-audit-log\") pod \"metrics-server-558488469d-rfr47\" (UID: \"199968f8-4c75-40d8-bc3c-2652edf4c7da\") " pod="openshift-monitoring/metrics-server-558488469d-rfr47"
Sep 17 17:35:34.480841 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:34.480762    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/199968f8-4c75-40d8-bc3c-2652edf4c7da-metrics-server-audit-profiles\") pod \"metrics-server-558488469d-rfr47\" (UID: \"199968f8-4c75-40d8-bc3c-2652edf4c7da\") " pod="openshift-monitoring/metrics-server-558488469d-rfr47"
Sep 17 17:35:34.480841 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:34.480802    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/199968f8-4c75-40d8-bc3c-2652edf4c7da-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-558488469d-rfr47\" (UID: \"199968f8-4c75-40d8-bc3c-2652edf4c7da\") " pod="openshift-monitoring/metrics-server-558488469d-rfr47"
Sep 17 17:35:34.480841 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:34.480827    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/199968f8-4c75-40d8-bc3c-2652edf4c7da-client-ca-bundle\") pod \"metrics-server-558488469d-rfr47\" (UID: \"199968f8-4c75-40d8-bc3c-2652edf4c7da\") " pod="openshift-monitoring/metrics-server-558488469d-rfr47"
Sep 17 17:35:34.481034 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:34.480885    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/199968f8-4c75-40d8-bc3c-2652edf4c7da-secret-metrics-server-client-certs\") pod \"metrics-server-558488469d-rfr47\" (UID: \"199968f8-4c75-40d8-bc3c-2652edf4c7da\") " pod="openshift-monitoring/metrics-server-558488469d-rfr47"
Sep 17 17:35:34.481034 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:34.480924    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pdqnm\" (UniqueName: \"kubernetes.io/projected/199968f8-4c75-40d8-bc3c-2652edf4c7da-kube-api-access-pdqnm\") pod \"metrics-server-558488469d-rfr47\" (UID: \"199968f8-4c75-40d8-bc3c-2652edf4c7da\") " pod="openshift-monitoring/metrics-server-558488469d-rfr47"
Sep 17 17:35:34.481340 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:34.481276    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/199968f8-4c75-40d8-bc3c-2652edf4c7da-audit-log\") pod \"metrics-server-558488469d-rfr47\" (UID: \"199968f8-4c75-40d8-bc3c-2652edf4c7da\") " pod="openshift-monitoring/metrics-server-558488469d-rfr47"
Sep 17 17:35:34.481670 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:34.481648    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/199968f8-4c75-40d8-bc3c-2652edf4c7da-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-558488469d-rfr47\" (UID: \"199968f8-4c75-40d8-bc3c-2652edf4c7da\") " pod="openshift-monitoring/metrics-server-558488469d-rfr47"
Sep 17 17:35:34.481920 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:34.481900    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/199968f8-4c75-40d8-bc3c-2652edf4c7da-metrics-server-audit-profiles\") pod \"metrics-server-558488469d-rfr47\" (UID: \"199968f8-4c75-40d8-bc3c-2652edf4c7da\") " pod="openshift-monitoring/metrics-server-558488469d-rfr47"
Sep 17 17:35:34.483269 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:34.483248    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/199968f8-4c75-40d8-bc3c-2652edf4c7da-secret-metrics-server-client-certs\") pod \"metrics-server-558488469d-rfr47\" (UID: \"199968f8-4c75-40d8-bc3c-2652edf4c7da\") " pod="openshift-monitoring/metrics-server-558488469d-rfr47"
Sep 17 17:35:34.483367 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:34.483315    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/199968f8-4c75-40d8-bc3c-2652edf4c7da-secret-metrics-server-tls\") pod \"metrics-server-558488469d-rfr47\" (UID: \"199968f8-4c75-40d8-bc3c-2652edf4c7da\") " pod="openshift-monitoring/metrics-server-558488469d-rfr47"
Sep 17 17:35:34.483367 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:34.483358    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/199968f8-4c75-40d8-bc3c-2652edf4c7da-client-ca-bundle\") pod \"metrics-server-558488469d-rfr47\" (UID: \"199968f8-4c75-40d8-bc3c-2652edf4c7da\") " pod="openshift-monitoring/metrics-server-558488469d-rfr47"
Sep 17 17:35:34.492255 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:34.492233    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pdqnm\" (UniqueName: \"kubernetes.io/projected/199968f8-4c75-40d8-bc3c-2652edf4c7da-kube-api-access-pdqnm\") pod \"metrics-server-558488469d-rfr47\" (UID: \"199968f8-4c75-40d8-bc3c-2652edf4c7da\") " pod="openshift-monitoring/metrics-server-558488469d-rfr47"
Sep 17 17:35:34.494304 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:34.494280    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-84c885658c-tzmkz" event={"ID":"1d665736-4c4e-49aa-ab27-8ab235e46063","Type":"ContainerStarted","Data":"26d4d9d87d41a7371fa2dcd754f713f1188a612e27db9470b2693c60970b1ad0"}
Sep 17 17:35:34.494401 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:34.494334    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-84c885658c-tzmkz" event={"ID":"1d665736-4c4e-49aa-ab27-8ab235e46063","Type":"ContainerStarted","Data":"499a66f4bcb0259297997ddb071fac76effba3ffda9ac8530ea3cd032cdad532"}
Sep 17 17:35:34.496517 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:34.496498    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-x5649" event={"ID":"70b48508-2726-4415-ba31-35246e0f6ad1","Type":"ContainerStarted","Data":"9ddd334d2d3e37736694f04c01cb986cf22b3e4685dc0917ca91170d5313ad47"}
Sep 17 17:35:34.496598 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:34.496537    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-x5649" event={"ID":"70b48508-2726-4415-ba31-35246e0f6ad1","Type":"ContainerStarted","Data":"8e724215c7302f3ac5986c10d2a7fef0b9010daa6f74e668928bc6fe0cbe79d3"}
Sep 17 17:35:34.544756 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:34.544708    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-84c885658c-tzmkz" podStartSLOduration=4.504248354 podStartE2EDuration="5.544691804s" totalImagesPullingTime="1.04044345s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:35:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:35:34.543739501 +0000 UTC m=+89.732613208" watchObservedRunningTime="2026-09-17 17:35:34.544691804 +0000 UTC m=+89.733565533"
Sep 17 17:35:34.545201 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:34.545183    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-558488469d-rfr47"
Sep 17 17:35:34.583768 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:34.583650    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-x5649" podStartSLOduration=4.708904948 podStartE2EDuration="5.58363096s" totalImagesPullingTime="804.700562ms" totalInitContainerRuntime="70.02545ms" isStatefulPod=true podCreationTimestamp="2026-09-17 17:35:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:35:34.583556589 +0000 UTC m=+89.772430318" watchObservedRunningTime="2026-09-17 17:35:34.58363096 +0000 UTC m=+89.772504668"
Sep 17 17:35:34.586462 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:34.586436    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j4ds_d3f553e8-b416-4d7d-bbef-991a0404b287/konnectivity-agent/0.log" limit=1024
Sep 17 17:35:34.671275 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:34.671245    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j4ds_d3f553e8-b416-4d7d-bbef-991a0404b287/konnectivity-agent/0.log" limit=1024
Sep 17 17:35:34.769260 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:34.769177    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j4ds_d3f553e8-b416-4d7d-bbef-991a0404b287/konnectivity-agent/0.log" limit=1024
Sep 17 17:35:34.813508 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:34.813487    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-558488469d-rfr47"]
Sep 17 17:35:34.815789 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:35:34.815766    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod199968f8_4c75_40d8_bc3c_2652edf4c7da.slice/crio-1ca951a588a6981a677000b2cc71afae7278bbd8bd3ff6a41a14f47250b9edb1 WatchSource:0}: Error finding container 1ca951a588a6981a677000b2cc71afae7278bbd8bd3ff6a41a14f47250b9edb1: Status 404 returned error can't find the container with id 1ca951a588a6981a677000b2cc71afae7278bbd8bd3ff6a41a14f47250b9edb1
Sep 17 17:35:35.502430 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:35.502390    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-558488469d-rfr47" event={"ID":"199968f8-4c75-40d8-bc3c-2652edf4c7da","Type":"ContainerStarted","Data":"1ca951a588a6981a677000b2cc71afae7278bbd8bd3ff6a41a14f47250b9edb1"}
Sep 17 17:35:36.217064 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:36.217016    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j4ds_d3f553e8-b416-4d7d-bbef-991a0404b287/konnectivity-agent/0.log" limit=1024
Sep 17 17:35:36.219051 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:36.219023    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 17 17:35:36.221471 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:36.221457    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:35:36.224361 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:36.224174    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 17 17:35:36.224361 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:36.224230    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 17 17:35:36.224361 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:36.224251    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 17 17:35:36.224361 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:36.224260    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-c46kfct83tn4k\""
Sep 17 17:35:36.224361 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:36.224266    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 17 17:35:36.224361 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:36.224275    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 17 17:35:36.224361 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:36.224302    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-6cbsr\""
Sep 17 17:35:36.224770 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:36.224604    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 17 17:35:36.224770 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:36.224639    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 17 17:35:36.224770 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:36.224647    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 17 17:35:36.224770 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:36.224723    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 17 17:35:36.225036 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:36.225022    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 17 17:35:36.225930 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:36.225907    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 17 17:35:36.226026 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:36.225928    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 17 17:35:36.232671 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:36.232634    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 17 17:35:36.235255 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:36.235237    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 17 17:35:36.241320 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:36.241303    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 17 17:35:36.284931 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:36.284908    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j4ds_d3f553e8-b416-4d7d-bbef-991a0404b287/konnectivity-agent/0.log" limit=1024
Sep 17 17:35:36.298998 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:36.298975    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/f2e735a3-7d6c-48d9-84a6-40e168ba42ea-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"f2e735a3-7d6c-48d9-84a6-40e168ba42ea\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:35:36.299114 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:36.299022    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f2e735a3-7d6c-48d9-84a6-40e168ba42ea-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f2e735a3-7d6c-48d9-84a6-40e168ba42ea\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:35:36.299114 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:36.299081    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f2e735a3-7d6c-48d9-84a6-40e168ba42ea-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"f2e735a3-7d6c-48d9-84a6-40e168ba42ea\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:35:36.299114 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:36.299106    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/f2e735a3-7d6c-48d9-84a6-40e168ba42ea-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"f2e735a3-7d6c-48d9-84a6-40e168ba42ea\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:35:36.299243 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:36.299130    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/f2e735a3-7d6c-48d9-84a6-40e168ba42ea-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"f2e735a3-7d6c-48d9-84a6-40e168ba42ea\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:35:36.299243 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:36.299181    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/f2e735a3-7d6c-48d9-84a6-40e168ba42ea-config\") pod \"prometheus-k8s-0\" (UID: \"f2e735a3-7d6c-48d9-84a6-40e168ba42ea\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:35:36.299308 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:36.299257    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f2e735a3-7d6c-48d9-84a6-40e168ba42ea-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f2e735a3-7d6c-48d9-84a6-40e168ba42ea\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:35:36.299308 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:36.299288    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f2e735a3-7d6c-48d9-84a6-40e168ba42ea-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"f2e735a3-7d6c-48d9-84a6-40e168ba42ea\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:35:36.299308 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:36.299304    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f2e735a3-7d6c-48d9-84a6-40e168ba42ea-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"f2e735a3-7d6c-48d9-84a6-40e168ba42ea\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:35:36.299445 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:36.299320    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f2e735a3-7d6c-48d9-84a6-40e168ba42ea-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f2e735a3-7d6c-48d9-84a6-40e168ba42ea\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:35:36.299445 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:36.299404    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f2e735a3-7d6c-48d9-84a6-40e168ba42ea-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"f2e735a3-7d6c-48d9-84a6-40e168ba42ea\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:35:36.299445 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:36.299429    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/f2e735a3-7d6c-48d9-84a6-40e168ba42ea-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"f2e735a3-7d6c-48d9-84a6-40e168ba42ea\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:35:36.299555 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:36.299460    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/f2e735a3-7d6c-48d9-84a6-40e168ba42ea-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"f2e735a3-7d6c-48d9-84a6-40e168ba42ea\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:35:36.299555 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:36.299510    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/f2e735a3-7d6c-48d9-84a6-40e168ba42ea-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"f2e735a3-7d6c-48d9-84a6-40e168ba42ea\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:35:36.299555 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:36.299538    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f2e735a3-7d6c-48d9-84a6-40e168ba42ea-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"f2e735a3-7d6c-48d9-84a6-40e168ba42ea\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:35:36.299654 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:36.299577    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/f2e735a3-7d6c-48d9-84a6-40e168ba42ea-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"f2e735a3-7d6c-48d9-84a6-40e168ba42ea\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:35:36.299654 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:36.299601    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/f2e735a3-7d6c-48d9-84a6-40e168ba42ea-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"f2e735a3-7d6c-48d9-84a6-40e168ba42ea\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:35:36.299654 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:36.299623    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9lsp2\" (UniqueName: \"kubernetes.io/projected/f2e735a3-7d6c-48d9-84a6-40e168ba42ea-kube-api-access-9lsp2\") pod \"prometheus-k8s-0\" (UID: \"f2e735a3-7d6c-48d9-84a6-40e168ba42ea\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:35:36.299771 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:36.299656    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f2e735a3-7d6c-48d9-84a6-40e168ba42ea-config-out\") pod \"prometheus-k8s-0\" (UID: \"f2e735a3-7d6c-48d9-84a6-40e168ba42ea\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:35:36.299771 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:36.299678    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f2e735a3-7d6c-48d9-84a6-40e168ba42ea-web-config\") pod \"prometheus-k8s-0\" (UID: \"f2e735a3-7d6c-48d9-84a6-40e168ba42ea\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:35:36.400547 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:36.400490    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f2e735a3-7d6c-48d9-84a6-40e168ba42ea-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"f2e735a3-7d6c-48d9-84a6-40e168ba42ea\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:35:36.400547 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:36.400517    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/f2e735a3-7d6c-48d9-84a6-40e168ba42ea-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"f2e735a3-7d6c-48d9-84a6-40e168ba42ea\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:35:36.400547 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:36.400534    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/f2e735a3-7d6c-48d9-84a6-40e168ba42ea-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"f2e735a3-7d6c-48d9-84a6-40e168ba42ea\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:35:36.400714 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:36.400550    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/f2e735a3-7d6c-48d9-84a6-40e168ba42ea-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"f2e735a3-7d6c-48d9-84a6-40e168ba42ea\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:35:36.400714 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:36.400564    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f2e735a3-7d6c-48d9-84a6-40e168ba42ea-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"f2e735a3-7d6c-48d9-84a6-40e168ba42ea\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:35:36.400714 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:36.400630    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/f2e735a3-7d6c-48d9-84a6-40e168ba42ea-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"f2e735a3-7d6c-48d9-84a6-40e168ba42ea\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:35:36.400714 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:36.400657    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/f2e735a3-7d6c-48d9-84a6-40e168ba42ea-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"f2e735a3-7d6c-48d9-84a6-40e168ba42ea\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:35:36.400714 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:36.400677    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9lsp2\" (UniqueName: \"kubernetes.io/projected/f2e735a3-7d6c-48d9-84a6-40e168ba42ea-kube-api-access-9lsp2\") pod \"prometheus-k8s-0\" (UID: \"f2e735a3-7d6c-48d9-84a6-40e168ba42ea\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:35:36.400714 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:36.400700    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f2e735a3-7d6c-48d9-84a6-40e168ba42ea-config-out\") pod \"prometheus-k8s-0\" (UID: \"f2e735a3-7d6c-48d9-84a6-40e168ba42ea\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:35:36.400985 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:36.400721    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f2e735a3-7d6c-48d9-84a6-40e168ba42ea-web-config\") pod \"prometheus-k8s-0\" (UID: \"f2e735a3-7d6c-48d9-84a6-40e168ba42ea\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:35:36.400985 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:36.400750    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/f2e735a3-7d6c-48d9-84a6-40e168ba42ea-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"f2e735a3-7d6c-48d9-84a6-40e168ba42ea\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:35:36.400985 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:36.400772    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f2e735a3-7d6c-48d9-84a6-40e168ba42ea-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f2e735a3-7d6c-48d9-84a6-40e168ba42ea\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:35:36.400985 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:36.400803    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f2e735a3-7d6c-48d9-84a6-40e168ba42ea-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"f2e735a3-7d6c-48d9-84a6-40e168ba42ea\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:35:36.400985 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:36.400826    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/f2e735a3-7d6c-48d9-84a6-40e168ba42ea-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"f2e735a3-7d6c-48d9-84a6-40e168ba42ea\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:35:36.400985 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:36.400850    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/f2e735a3-7d6c-48d9-84a6-40e168ba42ea-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"f2e735a3-7d6c-48d9-84a6-40e168ba42ea\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:35:36.400985 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:36.400881    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/f2e735a3-7d6c-48d9-84a6-40e168ba42ea-config\") pod \"prometheus-k8s-0\" (UID: \"f2e735a3-7d6c-48d9-84a6-40e168ba42ea\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:35:36.400985 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:36.400923    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f2e735a3-7d6c-48d9-84a6-40e168ba42ea-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f2e735a3-7d6c-48d9-84a6-40e168ba42ea\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:35:36.400985 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:36.400953    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f2e735a3-7d6c-48d9-84a6-40e168ba42ea-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"f2e735a3-7d6c-48d9-84a6-40e168ba42ea\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:35:36.400985 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:36.400974    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f2e735a3-7d6c-48d9-84a6-40e168ba42ea-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"f2e735a3-7d6c-48d9-84a6-40e168ba42ea\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:35:36.401503 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:36.401002    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f2e735a3-7d6c-48d9-84a6-40e168ba42ea-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f2e735a3-7d6c-48d9-84a6-40e168ba42ea\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:35:36.401503 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:36.401250    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/f2e735a3-7d6c-48d9-84a6-40e168ba42ea-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"f2e735a3-7d6c-48d9-84a6-40e168ba42ea\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:35:36.401503 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:36.401345    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/f2e735a3-7d6c-48d9-84a6-40e168ba42ea-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"f2e735a3-7d6c-48d9-84a6-40e168ba42ea\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:35:36.402026 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:36.401729    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f2e735a3-7d6c-48d9-84a6-40e168ba42ea-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f2e735a3-7d6c-48d9-84a6-40e168ba42ea\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:35:36.402842 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:36.402428    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/f2e735a3-7d6c-48d9-84a6-40e168ba42ea-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"f2e735a3-7d6c-48d9-84a6-40e168ba42ea\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:35:36.402842 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:36.401349    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f2e735a3-7d6c-48d9-84a6-40e168ba42ea-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"f2e735a3-7d6c-48d9-84a6-40e168ba42ea\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:35:36.404143 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:36.404098    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f2e735a3-7d6c-48d9-84a6-40e168ba42ea-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f2e735a3-7d6c-48d9-84a6-40e168ba42ea\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:35:36.404403 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:36.404373    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/f2e735a3-7d6c-48d9-84a6-40e168ba42ea-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"f2e735a3-7d6c-48d9-84a6-40e168ba42ea\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:35:36.404807 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:36.404782    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f2e735a3-7d6c-48d9-84a6-40e168ba42ea-config-out\") pod \"prometheus-k8s-0\" (UID: \"f2e735a3-7d6c-48d9-84a6-40e168ba42ea\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:35:36.404902 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:36.404801    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f2e735a3-7d6c-48d9-84a6-40e168ba42ea-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"f2e735a3-7d6c-48d9-84a6-40e168ba42ea\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:35:36.404902 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:36.404792    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f2e735a3-7d6c-48d9-84a6-40e168ba42ea-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f2e735a3-7d6c-48d9-84a6-40e168ba42ea\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:35:36.405654 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:36.405577    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/f2e735a3-7d6c-48d9-84a6-40e168ba42ea-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"f2e735a3-7d6c-48d9-84a6-40e168ba42ea\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:35:36.405654 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:36.405600    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/f2e735a3-7d6c-48d9-84a6-40e168ba42ea-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"f2e735a3-7d6c-48d9-84a6-40e168ba42ea\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:35:36.405654 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:36.405578    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/f2e735a3-7d6c-48d9-84a6-40e168ba42ea-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"f2e735a3-7d6c-48d9-84a6-40e168ba42ea\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:35:36.406543 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:36.406517    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/f2e735a3-7d6c-48d9-84a6-40e168ba42ea-config\") pod \"prometheus-k8s-0\" (UID: \"f2e735a3-7d6c-48d9-84a6-40e168ba42ea\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:35:36.406653 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:36.406597    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f2e735a3-7d6c-48d9-84a6-40e168ba42ea-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"f2e735a3-7d6c-48d9-84a6-40e168ba42ea\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:35:36.407067 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:36.407021    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f2e735a3-7d6c-48d9-84a6-40e168ba42ea-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"f2e735a3-7d6c-48d9-84a6-40e168ba42ea\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:35:36.407558 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:36.407534    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/f2e735a3-7d6c-48d9-84a6-40e168ba42ea-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"f2e735a3-7d6c-48d9-84a6-40e168ba42ea\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:35:36.407701 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:36.407685    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f2e735a3-7d6c-48d9-84a6-40e168ba42ea-web-config\") pod \"prometheus-k8s-0\" (UID: \"f2e735a3-7d6c-48d9-84a6-40e168ba42ea\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:35:36.408157 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:36.408141    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f2e735a3-7d6c-48d9-84a6-40e168ba42ea-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"f2e735a3-7d6c-48d9-84a6-40e168ba42ea\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:35:36.412755 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:36.412734    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9lsp2\" (UniqueName: \"kubernetes.io/projected/f2e735a3-7d6c-48d9-84a6-40e168ba42ea-kube-api-access-9lsp2\") pod \"prometheus-k8s-0\" (UID: \"f2e735a3-7d6c-48d9-84a6-40e168ba42ea\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:35:36.509035 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:36.508989    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-558488469d-rfr47" event={"ID":"199968f8-4c75-40d8-bc3c-2652edf4c7da","Type":"ContainerStarted","Data":"c35d773de0e12a13802019833e6bd3a16acfcf80688cdcad11ea148e4664e190"}
Sep 17 17:35:36.529876 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:36.529822    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-558488469d-rfr47" podStartSLOduration=1.244434296 podStartE2EDuration="2.529802086s" totalImagesPullingTime="1.28536779s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:35:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:35:36.528901083 +0000 UTC m=+91.717774790" watchObservedRunningTime="2026-09-17 17:35:36.529802086 +0000 UTC m=+91.718675792"
Sep 17 17:35:36.536231 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:36.536208    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:35:36.824729 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:36.824700    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 17 17:35:36.826577 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:35:36.826551    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2e735a3_7d6c_48d9_84a6_40e168ba42ea.slice/crio-9976a3830c20ca5f8c2c4f399f6151f4875fd25d57ebf475e0e50a5ee24e7edb WatchSource:0}: Error finding container 9976a3830c20ca5f8c2c4f399f6151f4875fd25d57ebf475e0e50a5ee24e7edb: Status 404 returned error can't find the container with id 9976a3830c20ca5f8c2c4f399f6151f4875fd25d57ebf475e0e50a5ee24e7edb
Sep 17 17:35:37.514038 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:37.513998    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f2e735a3-7d6c-48d9-84a6-40e168ba42ea","Type":"ContainerStarted","Data":"9976a3830c20ca5f8c2c4f399f6151f4875fd25d57ebf475e0e50a5ee24e7edb"}
Sep 17 17:35:37.984838 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:37.984803    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-dd5c969b-rmtb5"
Sep 17 17:35:37.985025 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:37.984868    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-dd5c969b-rmtb5"
Sep 17 17:35:37.989495 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:37.989474    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-dd5c969b-rmtb5"
Sep 17 17:35:38.399687 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:38.399655    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-dd5c969b-rmtb5"]
Sep 17 17:35:38.519245 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:38.519215    2574 generic.go:356] "Generic (PLEG): container finished" podID="f2e735a3-7d6c-48d9-84a6-40e168ba42ea" containerID="5bd1e9c20df384961796065ea0677bee6f4ed4c5932e6efbe47972a24370b8dd" exitCode=0
Sep 17 17:35:38.519614 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:38.519300    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f2e735a3-7d6c-48d9-84a6-40e168ba42ea","Type":"ContainerDied","Data":"5bd1e9c20df384961796065ea0677bee6f4ed4c5932e6efbe47972a24370b8dd"}
Sep 17 17:35:38.523212 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:38.523196    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-dd5c969b-rmtb5"
Sep 17 17:35:39.796165 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:39.796130    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j4ds_d3f553e8-b416-4d7d-bbef-991a0404b287/konnectivity-agent/0.log" limit=1024
Sep 17 17:35:39.897135 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:39.897101    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j4ds_d3f553e8-b416-4d7d-bbef-991a0404b287/konnectivity-agent/0.log" limit=1024
Sep 17 17:35:40.426518 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:40.426494    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-58dbcf67c5-dndpx"
Sep 17 17:35:40.623966 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:40.623935    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j4ds_d3f553e8-b416-4d7d-bbef-991a0404b287/konnectivity-agent/0.log" limit=1024
Sep 17 17:35:40.870807 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:40.870776    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j4ds_d3f553e8-b416-4d7d-bbef-991a0404b287/konnectivity-agent/0.log" limit=1024
Sep 17 17:35:41.535300 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:41.534600    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f2e735a3-7d6c-48d9-84a6-40e168ba42ea","Type":"ContainerStarted","Data":"84249ab289434482a1981728db3dca7d87005c61ceaa6610dfdb27b3741e2001"}
Sep 17 17:35:41.847384 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:41.847351    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-54df6d68d-464vf"]
Sep 17 17:35:41.849553 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:41.849536    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-54df6d68d-464vf"
Sep 17 17:35:41.858157 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:41.858135    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Sep 17 17:35:41.861688 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:41.861668    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-54df6d68d-464vf"]
Sep 17 17:35:41.952264 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:41.952237    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1a4569a8-f709-467c-b575-e7b6defddf64-trusted-ca-bundle\") pod \"console-54df6d68d-464vf\" (UID: \"1a4569a8-f709-467c-b575-e7b6defddf64\") " pod="openshift-console/console-54df6d68d-464vf"
Sep 17 17:35:41.952605 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:41.952267    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1a4569a8-f709-467c-b575-e7b6defddf64-service-ca\") pod \"console-54df6d68d-464vf\" (UID: \"1a4569a8-f709-467c-b575-e7b6defddf64\") " pod="openshift-console/console-54df6d68d-464vf"
Sep 17 17:35:41.952605 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:41.952289    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vmr47\" (UniqueName: \"kubernetes.io/projected/1a4569a8-f709-467c-b575-e7b6defddf64-kube-api-access-vmr47\") pod \"console-54df6d68d-464vf\" (UID: \"1a4569a8-f709-467c-b575-e7b6defddf64\") " pod="openshift-console/console-54df6d68d-464vf"
Sep 17 17:35:41.952605 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:41.952319    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1a4569a8-f709-467c-b575-e7b6defddf64-console-oauth-config\") pod \"console-54df6d68d-464vf\" (UID: \"1a4569a8-f709-467c-b575-e7b6defddf64\") " pod="openshift-console/console-54df6d68d-464vf"
Sep 17 17:35:41.952605 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:41.952369    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1a4569a8-f709-467c-b575-e7b6defddf64-console-config\") pod \"console-54df6d68d-464vf\" (UID: \"1a4569a8-f709-467c-b575-e7b6defddf64\") " pod="openshift-console/console-54df6d68d-464vf"
Sep 17 17:35:41.952605 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:41.952389    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1a4569a8-f709-467c-b575-e7b6defddf64-console-serving-cert\") pod \"console-54df6d68d-464vf\" (UID: \"1a4569a8-f709-467c-b575-e7b6defddf64\") " pod="openshift-console/console-54df6d68d-464vf"
Sep 17 17:35:41.952605 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:41.952426    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1a4569a8-f709-467c-b575-e7b6defddf64-oauth-serving-cert\") pod \"console-54df6d68d-464vf\" (UID: \"1a4569a8-f709-467c-b575-e7b6defddf64\") " pod="openshift-console/console-54df6d68d-464vf"
Sep 17 17:35:42.053505 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:42.053476    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1a4569a8-f709-467c-b575-e7b6defddf64-oauth-serving-cert\") pod \"console-54df6d68d-464vf\" (UID: \"1a4569a8-f709-467c-b575-e7b6defddf64\") " pod="openshift-console/console-54df6d68d-464vf"
Sep 17 17:35:42.053622 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:42.053510    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1a4569a8-f709-467c-b575-e7b6defddf64-trusted-ca-bundle\") pod \"console-54df6d68d-464vf\" (UID: \"1a4569a8-f709-467c-b575-e7b6defddf64\") " pod="openshift-console/console-54df6d68d-464vf"
Sep 17 17:35:42.053622 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:42.053529    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1a4569a8-f709-467c-b575-e7b6defddf64-service-ca\") pod \"console-54df6d68d-464vf\" (UID: \"1a4569a8-f709-467c-b575-e7b6defddf64\") " pod="openshift-console/console-54df6d68d-464vf"
Sep 17 17:35:42.053622 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:42.053555    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vmr47\" (UniqueName: \"kubernetes.io/projected/1a4569a8-f709-467c-b575-e7b6defddf64-kube-api-access-vmr47\") pod \"console-54df6d68d-464vf\" (UID: \"1a4569a8-f709-467c-b575-e7b6defddf64\") " pod="openshift-console/console-54df6d68d-464vf"
Sep 17 17:35:42.053773 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:42.053676    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1a4569a8-f709-467c-b575-e7b6defddf64-console-oauth-config\") pod \"console-54df6d68d-464vf\" (UID: \"1a4569a8-f709-467c-b575-e7b6defddf64\") " pod="openshift-console/console-54df6d68d-464vf"
Sep 17 17:35:42.053827 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:42.053776    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1a4569a8-f709-467c-b575-e7b6defddf64-console-config\") pod \"console-54df6d68d-464vf\" (UID: \"1a4569a8-f709-467c-b575-e7b6defddf64\") " pod="openshift-console/console-54df6d68d-464vf"
Sep 17 17:35:42.053827 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:42.053802    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1a4569a8-f709-467c-b575-e7b6defddf64-console-serving-cert\") pod \"console-54df6d68d-464vf\" (UID: \"1a4569a8-f709-467c-b575-e7b6defddf64\") " pod="openshift-console/console-54df6d68d-464vf"
Sep 17 17:35:42.054324 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:42.054305    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1a4569a8-f709-467c-b575-e7b6defddf64-oauth-serving-cert\") pod \"console-54df6d68d-464vf\" (UID: \"1a4569a8-f709-467c-b575-e7b6defddf64\") " pod="openshift-console/console-54df6d68d-464vf"
Sep 17 17:35:42.054421 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:42.054305    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1a4569a8-f709-467c-b575-e7b6defddf64-service-ca\") pod \"console-54df6d68d-464vf\" (UID: \"1a4569a8-f709-467c-b575-e7b6defddf64\") " pod="openshift-console/console-54df6d68d-464vf"
Sep 17 17:35:42.054492 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:42.054459    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1a4569a8-f709-467c-b575-e7b6defddf64-console-config\") pod \"console-54df6d68d-464vf\" (UID: \"1a4569a8-f709-467c-b575-e7b6defddf64\") " pod="openshift-console/console-54df6d68d-464vf"
Sep 17 17:35:42.054607 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:42.054584    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1a4569a8-f709-467c-b575-e7b6defddf64-trusted-ca-bundle\") pod \"console-54df6d68d-464vf\" (UID: \"1a4569a8-f709-467c-b575-e7b6defddf64\") " pod="openshift-console/console-54df6d68d-464vf"
Sep 17 17:35:42.055990 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:42.055973    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1a4569a8-f709-467c-b575-e7b6defddf64-console-oauth-config\") pod \"console-54df6d68d-464vf\" (UID: \"1a4569a8-f709-467c-b575-e7b6defddf64\") " pod="openshift-console/console-54df6d68d-464vf"
Sep 17 17:35:42.056158 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:42.056143    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1a4569a8-f709-467c-b575-e7b6defddf64-console-serving-cert\") pod \"console-54df6d68d-464vf\" (UID: \"1a4569a8-f709-467c-b575-e7b6defddf64\") " pod="openshift-console/console-54df6d68d-464vf"
Sep 17 17:35:42.066314 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:42.066289    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vmr47\" (UniqueName: \"kubernetes.io/projected/1a4569a8-f709-467c-b575-e7b6defddf64-kube-api-access-vmr47\") pod \"console-54df6d68d-464vf\" (UID: \"1a4569a8-f709-467c-b575-e7b6defddf64\") " pod="openshift-console/console-54df6d68d-464vf"
Sep 17 17:35:42.159945 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:42.159868    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-54df6d68d-464vf"
Sep 17 17:35:42.438704 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:42.438635    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-54df6d68d-464vf"]
Sep 17 17:35:42.441985 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:35:42.441953    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a4569a8_f709_467c_b575_e7b6defddf64.slice/crio-b2f3531e73c75098665a4d01f88cd92894c91c4179b65c1b5421eceeee8190f7 WatchSource:0}: Error finding container b2f3531e73c75098665a4d01f88cd92894c91c4179b65c1b5421eceeee8190f7: Status 404 returned error can't find the container with id b2f3531e73c75098665a4d01f88cd92894c91c4179b65c1b5421eceeee8190f7
Sep 17 17:35:42.542768 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:42.542654    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f2e735a3-7d6c-48d9-84a6-40e168ba42ea","Type":"ContainerStarted","Data":"47c3256f8159b4161eaf97423fb425cc73bd0e8d6a4cfdee1430036f912bd74c"}
Sep 17 17:35:42.544834 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:42.544362    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-54df6d68d-464vf" event={"ID":"1a4569a8-f709-467c-b575-e7b6defddf64","Type":"ContainerStarted","Data":"9ee1b989c83817e8ec3288fa7d9f33c43449c4f94d2e27dfda5a9e4d61951016"}
Sep 17 17:35:42.544834 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:42.544422    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-54df6d68d-464vf" event={"ID":"1a4569a8-f709-467c-b575-e7b6defddf64","Type":"ContainerStarted","Data":"b2f3531e73c75098665a4d01f88cd92894c91c4179b65c1b5421eceeee8190f7"}
Sep 17 17:35:42.569947 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:42.569889    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-54df6d68d-464vf" podStartSLOduration=1.569868063 podStartE2EDuration="1.569868063s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:35:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:35:42.56899321 +0000 UTC m=+97.757866916" watchObservedRunningTime="2026-09-17 17:35:42.569868063 +0000 UTC m=+97.758741786"
Sep 17 17:35:43.552023 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:43.551991    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f2e735a3-7d6c-48d9-84a6-40e168ba42ea","Type":"ContainerStarted","Data":"ebf8d2f40362e72d769e0b7eb57645ce6f65cae04736275dc33e9f8d214f6a8c"}
Sep 17 17:35:43.552448 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:43.552073    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f2e735a3-7d6c-48d9-84a6-40e168ba42ea","Type":"ContainerStarted","Data":"081b3c8c9915f35abc53be444108613dcdfd46bb4e1b5a318388f908b2273777"}
Sep 17 17:35:43.552448 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:43.552107    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f2e735a3-7d6c-48d9-84a6-40e168ba42ea","Type":"ContainerStarted","Data":"b48ba3b5af18adb7ebac1e07dd5923d84ca016bd2667598ac6cfc7de6e6849d4"}
Sep 17 17:35:43.552448 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:43.552136    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f2e735a3-7d6c-48d9-84a6-40e168ba42ea","Type":"ContainerStarted","Data":"d808f041a38de295686a2cd50b9d5e3ec2f7bb08b97e870717a74238c3b1ce1b"}
Sep 17 17:35:43.586822 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:43.586523    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.07285121 podStartE2EDuration="7.586501975s" totalImagesPullingTime="5.495820186s" totalInitContainerRuntime="17.830579ms" isStatefulPod=false podCreationTimestamp="2026-09-17 17:35:36 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:35:43.584775624 +0000 UTC m=+98.773649340" watchObservedRunningTime="2026-09-17 17:35:43.586501975 +0000 UTC m=+98.775375683"
Sep 17 17:35:44.396986 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:44.396952    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-j9d2w"
Sep 17 17:35:46.537086 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:46.537037    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:35:47.446330 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:47.446296    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j4ds_d3f553e8-b416-4d7d-bbef-991a0404b287/konnectivity-agent/0.log" limit=1024
Sep 17 17:35:47.546370 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:47.546309    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j4ds_d3f553e8-b416-4d7d-bbef-991a0404b287/konnectivity-agent/0.log" limit=1024
Sep 17 17:35:49.190704 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:49.190676    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j4ds_d3f553e8-b416-4d7d-bbef-991a0404b287/konnectivity-agent/0.log" limit=1024
Sep 17 17:35:49.298269 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:49.298235    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j4ds_d3f553e8-b416-4d7d-bbef-991a0404b287/konnectivity-agent/0.log" limit=1024
Sep 17 17:35:49.388366 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:49.388329    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j4ds_d3f553e8-b416-4d7d-bbef-991a0404b287/konnectivity-agent/0.log" limit=1024
Sep 17 17:35:49.491169 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:49.491144    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j4ds_d3f553e8-b416-4d7d-bbef-991a0404b287/konnectivity-agent/0.log" limit=1024
Sep 17 17:35:49.589358 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:49.589331    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j4ds_d3f553e8-b416-4d7d-bbef-991a0404b287/konnectivity-agent/0.log" limit=1024
Sep 17 17:35:49.833369 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:49.833293    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j4ds_d3f553e8-b416-4d7d-bbef-991a0404b287/konnectivity-agent/0.log" limit=1024
Sep 17 17:35:50.042312 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:50.042284    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j4ds_d3f553e8-b416-4d7d-bbef-991a0404b287/konnectivity-agent/0.log" limit=1024
Sep 17 17:35:51.016746 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:51.016709    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j4ds_d3f553e8-b416-4d7d-bbef-991a0404b287/konnectivity-agent/0.log" limit=1024
Sep 17 17:35:51.217674 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:51.217638    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j4ds_d3f553e8-b416-4d7d-bbef-991a0404b287/konnectivity-agent/0.log" limit=1024
Sep 17 17:35:52.160429 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:52.160393    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-54df6d68d-464vf"
Sep 17 17:35:52.160854 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:52.160505    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-54df6d68d-464vf"
Sep 17 17:35:52.165592 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:52.165572    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-54df6d68d-464vf"
Sep 17 17:35:52.589424 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:52.589398    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-54df6d68d-464vf"
Sep 17 17:35:52.806695 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:52.806663    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j4ds_d3f553e8-b416-4d7d-bbef-991a0404b287/konnectivity-agent/0.log" limit=1024
Sep 17 17:35:52.903027 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:52.902962    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j4ds_d3f553e8-b416-4d7d-bbef-991a0404b287/konnectivity-agent/0.log" limit=1024
Sep 17 17:35:53.012392 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:53.012364    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j4ds_d3f553e8-b416-4d7d-bbef-991a0404b287/konnectivity-agent/0.log" limit=1024
Sep 17 17:35:53.104569 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:53.104542    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j4ds_d3f553e8-b416-4d7d-bbef-991a0404b287/konnectivity-agent/0.log" limit=1024
Sep 17 17:35:54.546038 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:54.546009    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-558488469d-rfr47"
Sep 17 17:35:54.546450 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:54.546093    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-558488469d-rfr47"
Sep 17 17:35:56.613596 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:56.613565    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j4ds_d3f553e8-b416-4d7d-bbef-991a0404b287/konnectivity-agent/0.log" limit=1024
Sep 17 17:35:56.715329 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:56.715298    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j4ds_d3f553e8-b416-4d7d-bbef-991a0404b287/konnectivity-agent/0.log" limit=1024
Sep 17 17:35:58.550989 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:58.550955    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j4ds_d3f553e8-b416-4d7d-bbef-991a0404b287/konnectivity-agent/0.log" limit=1024
Sep 17 17:35:58.660908 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:35:58.660882    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j4ds_d3f553e8-b416-4d7d-bbef-991a0404b287/konnectivity-agent/0.log" limit=1024
Sep 17 17:36:02.487095 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:02.487058    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j4ds_d3f553e8-b416-4d7d-bbef-991a0404b287/konnectivity-agent/0.log" limit=1024
Sep 17 17:36:02.579399 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:02.579368    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j4ds_d3f553e8-b416-4d7d-bbef-991a0404b287/konnectivity-agent/0.log" limit=1024
Sep 17 17:36:04.544840 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:04.544804    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-dd5c969b-rmtb5" podUID="035b396a-6ac7-4fe0-a639-104d2ff1bad3" containerName="console" containerID="cri-o://268e15ec302482efe760f1076432389c849834d3885279743f08141e998eb8ed" gracePeriod=15
Sep 17 17:36:05.316731 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:05.316705    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-dd5c969b-rmtb5"
Sep 17 17:36:05.356524 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:05.356498    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-trl25\" (UniqueName: \"kubernetes.io/projected/035b396a-6ac7-4fe0-a639-104d2ff1bad3-kube-api-access-trl25\") pod \"035b396a-6ac7-4fe0-a639-104d2ff1bad3\" (UID: \"035b396a-6ac7-4fe0-a639-104d2ff1bad3\") "
Sep 17 17:36:05.356679 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:05.356528    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/035b396a-6ac7-4fe0-a639-104d2ff1bad3-console-oauth-config\") pod \"035b396a-6ac7-4fe0-a639-104d2ff1bad3\" (UID: \"035b396a-6ac7-4fe0-a639-104d2ff1bad3\") "
Sep 17 17:36:05.356679 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:05.356575    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/035b396a-6ac7-4fe0-a639-104d2ff1bad3-service-ca\") pod \"035b396a-6ac7-4fe0-a639-104d2ff1bad3\" (UID: \"035b396a-6ac7-4fe0-a639-104d2ff1bad3\") "
Sep 17 17:36:05.356679 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:05.356597    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/035b396a-6ac7-4fe0-a639-104d2ff1bad3-console-serving-cert\") pod \"035b396a-6ac7-4fe0-a639-104d2ff1bad3\" (UID: \"035b396a-6ac7-4fe0-a639-104d2ff1bad3\") "
Sep 17 17:36:05.356679 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:05.356627    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/035b396a-6ac7-4fe0-a639-104d2ff1bad3-console-config\") pod \"035b396a-6ac7-4fe0-a639-104d2ff1bad3\" (UID: \"035b396a-6ac7-4fe0-a639-104d2ff1bad3\") "
Sep 17 17:36:05.356679 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:05.356668    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/035b396a-6ac7-4fe0-a639-104d2ff1bad3-oauth-serving-cert\") pod \"035b396a-6ac7-4fe0-a639-104d2ff1bad3\" (UID: \"035b396a-6ac7-4fe0-a639-104d2ff1bad3\") "
Sep 17 17:36:05.357155 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:05.357119    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/035b396a-6ac7-4fe0-a639-104d2ff1bad3-service-ca" (OuterVolumeSpecName: "service-ca") pod "035b396a-6ac7-4fe0-a639-104d2ff1bad3" (UID: "035b396a-6ac7-4fe0-a639-104d2ff1bad3"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:36:05.357155 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:05.357132    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/035b396a-6ac7-4fe0-a639-104d2ff1bad3-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "035b396a-6ac7-4fe0-a639-104d2ff1bad3" (UID: "035b396a-6ac7-4fe0-a639-104d2ff1bad3"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:36:05.357324 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:05.357127    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/035b396a-6ac7-4fe0-a639-104d2ff1bad3-console-config" (OuterVolumeSpecName: "console-config") pod "035b396a-6ac7-4fe0-a639-104d2ff1bad3" (UID: "035b396a-6ac7-4fe0-a639-104d2ff1bad3"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:36:05.358870 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:05.358842    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/035b396a-6ac7-4fe0-a639-104d2ff1bad3-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "035b396a-6ac7-4fe0-a639-104d2ff1bad3" (UID: "035b396a-6ac7-4fe0-a639-104d2ff1bad3"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:36:05.358870 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:05.358861    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/035b396a-6ac7-4fe0-a639-104d2ff1bad3-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "035b396a-6ac7-4fe0-a639-104d2ff1bad3" (UID: "035b396a-6ac7-4fe0-a639-104d2ff1bad3"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:36:05.359140 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:05.359118    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/035b396a-6ac7-4fe0-a639-104d2ff1bad3-kube-api-access-trl25" (OuterVolumeSpecName: "kube-api-access-trl25") pod "035b396a-6ac7-4fe0-a639-104d2ff1bad3" (UID: "035b396a-6ac7-4fe0-a639-104d2ff1bad3"). InnerVolumeSpecName "kube-api-access-trl25". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:36:05.457368 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:05.457285    2574 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/035b396a-6ac7-4fe0-a639-104d2ff1bad3-service-ca\") on node \"ip-10-0-134-96.ec2.internal\" DevicePath \"\""
Sep 17 17:36:05.457368 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:05.457313    2574 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/035b396a-6ac7-4fe0-a639-104d2ff1bad3-console-serving-cert\") on node \"ip-10-0-134-96.ec2.internal\" DevicePath \"\""
Sep 17 17:36:05.457368 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:05.457326    2574 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/035b396a-6ac7-4fe0-a639-104d2ff1bad3-console-config\") on node \"ip-10-0-134-96.ec2.internal\" DevicePath \"\""
Sep 17 17:36:05.457368 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:05.457340    2574 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/035b396a-6ac7-4fe0-a639-104d2ff1bad3-oauth-serving-cert\") on node \"ip-10-0-134-96.ec2.internal\" DevicePath \"\""
Sep 17 17:36:05.457368 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:05.457350    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-trl25\" (UniqueName: \"kubernetes.io/projected/035b396a-6ac7-4fe0-a639-104d2ff1bad3-kube-api-access-trl25\") on node \"ip-10-0-134-96.ec2.internal\" DevicePath \"\""
Sep 17 17:36:05.457368 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:05.457364    2574 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/035b396a-6ac7-4fe0-a639-104d2ff1bad3-console-oauth-config\") on node \"ip-10-0-134-96.ec2.internal\" DevicePath \"\""
Sep 17 17:36:05.636007 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:05.635974    2574 generic.go:356] "Generic (PLEG): container finished" podID="035b396a-6ac7-4fe0-a639-104d2ff1bad3" containerID="268e15ec302482efe760f1076432389c849834d3885279743f08141e998eb8ed" exitCode=0
Sep 17 17:36:05.636438 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:05.636026    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-dd5c969b-rmtb5"
Sep 17 17:36:05.636438 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:05.636072    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-dd5c969b-rmtb5" event={"ID":"035b396a-6ac7-4fe0-a639-104d2ff1bad3","Type":"ContainerDied","Data":"268e15ec302482efe760f1076432389c849834d3885279743f08141e998eb8ed"}
Sep 17 17:36:05.636438 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:05.636122    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-dd5c969b-rmtb5" event={"ID":"035b396a-6ac7-4fe0-a639-104d2ff1bad3","Type":"ContainerDied","Data":"87d379760ebda8e3c8eb10796b2913d359334ebd3ff43da95547299e20f74b45"}
Sep 17 17:36:05.636438 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:05.636141    2574 scope.go:117] "RemoveContainer" containerID="268e15ec302482efe760f1076432389c849834d3885279743f08141e998eb8ed"
Sep 17 17:36:05.644550 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:05.644534    2574 scope.go:117] "RemoveContainer" containerID="268e15ec302482efe760f1076432389c849834d3885279743f08141e998eb8ed"
Sep 17 17:36:05.644816 ip-10-0-134-96 kubenswrapper[2574]: E0917 17:36:05.644791    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"268e15ec302482efe760f1076432389c849834d3885279743f08141e998eb8ed\": container with ID starting with 268e15ec302482efe760f1076432389c849834d3885279743f08141e998eb8ed not found: ID does not exist" containerID="268e15ec302482efe760f1076432389c849834d3885279743f08141e998eb8ed"
Sep 17 17:36:05.644883 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:05.644828    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"268e15ec302482efe760f1076432389c849834d3885279743f08141e998eb8ed"} err="failed to get container status \"268e15ec302482efe760f1076432389c849834d3885279743f08141e998eb8ed\": rpc error: code = NotFound desc = could not find container \"268e15ec302482efe760f1076432389c849834d3885279743f08141e998eb8ed\": container with ID starting with 268e15ec302482efe760f1076432389c849834d3885279743f08141e998eb8ed not found: ID does not exist"
Sep 17 17:36:05.660103 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:05.660085    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-dd5c969b-rmtb5"]
Sep 17 17:36:05.677268 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:05.677248    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-dd5c969b-rmtb5"]
Sep 17 17:36:06.954796 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:06.954765    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="035b396a-6ac7-4fe0-a639-104d2ff1bad3" path="/var/lib/kubelet/pods/035b396a-6ac7-4fe0-a639-104d2ff1bad3/volumes"
Sep 17 17:36:07.134641 ip-10-0-134-96 kubenswrapper[2574]: E0917 17:36:07.134612    2574 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0d18822_7421_4563_8c4d_777f26ae6c0e.slice/crio-06c8cc6f64e7d77b30ad6545da8eb495db3f3105558f0b3657b7908e7165fe51.scope\": RecentStats: unable to find data in memory cache]"
Sep 17 17:36:07.134751 ip-10-0-134-96 kubenswrapper[2574]: E0917 17:36:07.134685    2574 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0d18822_7421_4563_8c4d_777f26ae6c0e.slice/crio-06c8cc6f64e7d77b30ad6545da8eb495db3f3105558f0b3657b7908e7165fe51.scope\": RecentStats: unable to find data in memory cache]"
Sep 17 17:36:07.645846 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:07.645813    2574 generic.go:356] "Generic (PLEG): container finished" podID="d0d18822-7421-4563-8c4d-777f26ae6c0e" containerID="06c8cc6f64e7d77b30ad6545da8eb495db3f3105558f0b3657b7908e7165fe51" exitCode=0
Sep 17 17:36:07.646006 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:07.645853    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-6f9c644dd9-d8vvv" event={"ID":"d0d18822-7421-4563-8c4d-777f26ae6c0e","Type":"ContainerDied","Data":"06c8cc6f64e7d77b30ad6545da8eb495db3f3105558f0b3657b7908e7165fe51"}
Sep 17 17:36:07.646265 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:07.646247    2574 scope.go:117] "RemoveContainer" containerID="06c8cc6f64e7d77b30ad6545da8eb495db3f3105558f0b3657b7908e7165fe51"
Sep 17 17:36:08.651935 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:08.651902    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-6f9c644dd9-d8vvv" event={"ID":"d0d18822-7421-4563-8c4d-777f26ae6c0e","Type":"ContainerStarted","Data":"cbb774ada3b872ca2d44dae04a4d94d8d42a059cf3da85cfdb0cdcf8dc7c2555"}
Sep 17 17:36:12.670272 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:12.670239    2574 generic.go:356] "Generic (PLEG): container finished" podID="88497892-29d0-4b92-803b-2ce9ebebf402" containerID="2210b41930ca6c286cd8ee9c2380485d3323d99ab9ad7d12a294dfc5a896214d" exitCode=0
Sep 17 17:36:12.670634 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:12.670310    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-ns2j6" event={"ID":"88497892-29d0-4b92-803b-2ce9ebebf402","Type":"ContainerDied","Data":"2210b41930ca6c286cd8ee9c2380485d3323d99ab9ad7d12a294dfc5a896214d"}
Sep 17 17:36:12.670634 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:12.670600    2574 scope.go:117] "RemoveContainer" containerID="2210b41930ca6c286cd8ee9c2380485d3323d99ab9ad7d12a294dfc5a896214d"
Sep 17 17:36:13.559934 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:13.559893    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j4ds_d3f553e8-b416-4d7d-bbef-991a0404b287/konnectivity-agent/0.log" limit=1024
Sep 17 17:36:13.675502 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:13.675471    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-ns2j6" event={"ID":"88497892-29d0-4b92-803b-2ce9ebebf402","Type":"ContainerStarted","Data":"a9468829f2efb06fe9deb59b9b857be291d2710899489daddcb7dde3cec2b4ee"}
Sep 17 17:36:13.705444 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:13.705416    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j4ds_d3f553e8-b416-4d7d-bbef-991a0404b287/konnectivity-agent/0.log" limit=1024
Sep 17 17:36:14.550587 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:14.550554    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-558488469d-rfr47"
Sep 17 17:36:14.554241 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:14.554220    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-558488469d-rfr47"
Sep 17 17:36:22.714389 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:22.714356    2574 generic.go:356] "Generic (PLEG): container finished" podID="b7aea158-3b08-4631-93f5-c03c3dfb7998" containerID="12c35d710b9ed8098b09894dd6b614a82ce7cb3539631e9985c3e298f5aafc48" exitCode=0
Sep 17 17:36:22.714789 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:22.714436    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-gvwc6" event={"ID":"b7aea158-3b08-4631-93f5-c03c3dfb7998","Type":"ContainerDied","Data":"12c35d710b9ed8098b09894dd6b614a82ce7cb3539631e9985c3e298f5aafc48"}
Sep 17 17:36:22.714789 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:22.714708    2574 scope.go:117] "RemoveContainer" containerID="12c35d710b9ed8098b09894dd6b614a82ce7cb3539631e9985c3e298f5aafc48"
Sep 17 17:36:23.725467 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:23.725431    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-gvwc6" event={"ID":"b7aea158-3b08-4631-93f5-c03c3dfb7998","Type":"ContainerStarted","Data":"646a3d1f25a9ea8fe4f69893442fe65ee818d295561e9cf5b8158cd6093bcf73"}
Sep 17 17:36:36.537271 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:36.537240    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:36:36.558161 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:36.558136    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:36:36.786695 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:36.786668    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:36:45.954009 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:45.953978    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j4ds_d3f553e8-b416-4d7d-bbef-991a0404b287/konnectivity-agent/0.log" limit=1024
Sep 17 17:36:46.043400 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:46.043371    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j4ds_d3f553e8-b416-4d7d-bbef-991a0404b287/konnectivity-agent/0.log" limit=1024
Sep 17 17:36:54.538360 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:54.538287    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 17 17:36:54.538836 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:54.538611    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="f2e735a3-7d6c-48d9-84a6-40e168ba42ea" containerName="prometheus" containerID="cri-o://84249ab289434482a1981728db3dca7d87005c61ceaa6610dfdb27b3741e2001" gracePeriod=600
Sep 17 17:36:54.538836 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:54.538631    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="f2e735a3-7d6c-48d9-84a6-40e168ba42ea" containerName="kube-rbac-proxy" containerID="cri-o://081b3c8c9915f35abc53be444108613dcdfd46bb4e1b5a318388f908b2273777" gracePeriod=600
Sep 17 17:36:54.538836 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:54.538666    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="f2e735a3-7d6c-48d9-84a6-40e168ba42ea" containerName="thanos-sidecar" containerID="cri-o://d808f041a38de295686a2cd50b9d5e3ec2f7bb08b97e870717a74238c3b1ce1b" gracePeriod=600
Sep 17 17:36:54.538836 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:54.538682    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="f2e735a3-7d6c-48d9-84a6-40e168ba42ea" containerName="kube-rbac-proxy-thanos" containerID="cri-o://ebf8d2f40362e72d769e0b7eb57645ce6f65cae04736275dc33e9f8d214f6a8c" gracePeriod=600
Sep 17 17:36:54.538836 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:54.538687    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="f2e735a3-7d6c-48d9-84a6-40e168ba42ea" containerName="kube-rbac-proxy-web" containerID="cri-o://b48ba3b5af18adb7ebac1e07dd5923d84ca016bd2667598ac6cfc7de6e6849d4" gracePeriod=600
Sep 17 17:36:54.538836 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:54.538682    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="f2e735a3-7d6c-48d9-84a6-40e168ba42ea" containerName="config-reloader" containerID="cri-o://47c3256f8159b4161eaf97423fb425cc73bd0e8d6a4cfdee1430036f912bd74c" gracePeriod=600
Sep 17 17:36:54.766374 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:54.766350    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:36:54.842617 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:54.842532    2574 generic.go:356] "Generic (PLEG): container finished" podID="f2e735a3-7d6c-48d9-84a6-40e168ba42ea" containerID="ebf8d2f40362e72d769e0b7eb57645ce6f65cae04736275dc33e9f8d214f6a8c" exitCode=0
Sep 17 17:36:54.842617 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:54.842557    2574 generic.go:356] "Generic (PLEG): container finished" podID="f2e735a3-7d6c-48d9-84a6-40e168ba42ea" containerID="081b3c8c9915f35abc53be444108613dcdfd46bb4e1b5a318388f908b2273777" exitCode=0
Sep 17 17:36:54.842617 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:54.842563    2574 generic.go:356] "Generic (PLEG): container finished" podID="f2e735a3-7d6c-48d9-84a6-40e168ba42ea" containerID="b48ba3b5af18adb7ebac1e07dd5923d84ca016bd2667598ac6cfc7de6e6849d4" exitCode=0
Sep 17 17:36:54.842617 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:54.842569    2574 generic.go:356] "Generic (PLEG): container finished" podID="f2e735a3-7d6c-48d9-84a6-40e168ba42ea" containerID="d808f041a38de295686a2cd50b9d5e3ec2f7bb08b97e870717a74238c3b1ce1b" exitCode=0
Sep 17 17:36:54.842617 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:54.842574    2574 generic.go:356] "Generic (PLEG): container finished" podID="f2e735a3-7d6c-48d9-84a6-40e168ba42ea" containerID="47c3256f8159b4161eaf97423fb425cc73bd0e8d6a4cfdee1430036f912bd74c" exitCode=0
Sep 17 17:36:54.842617 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:54.842579    2574 generic.go:356] "Generic (PLEG): container finished" podID="f2e735a3-7d6c-48d9-84a6-40e168ba42ea" containerID="84249ab289434482a1981728db3dca7d87005c61ceaa6610dfdb27b3741e2001" exitCode=0
Sep 17 17:36:54.842617 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:54.842610    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f2e735a3-7d6c-48d9-84a6-40e168ba42ea","Type":"ContainerDied","Data":"ebf8d2f40362e72d769e0b7eb57645ce6f65cae04736275dc33e9f8d214f6a8c"}
Sep 17 17:36:54.843037 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:54.842630    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:36:54.843037 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:54.842685    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f2e735a3-7d6c-48d9-84a6-40e168ba42ea","Type":"ContainerDied","Data":"081b3c8c9915f35abc53be444108613dcdfd46bb4e1b5a318388f908b2273777"}
Sep 17 17:36:54.843037 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:54.842712    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f2e735a3-7d6c-48d9-84a6-40e168ba42ea","Type":"ContainerDied","Data":"b48ba3b5af18adb7ebac1e07dd5923d84ca016bd2667598ac6cfc7de6e6849d4"}
Sep 17 17:36:54.843037 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:54.842737    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f2e735a3-7d6c-48d9-84a6-40e168ba42ea","Type":"ContainerDied","Data":"d808f041a38de295686a2cd50b9d5e3ec2f7bb08b97e870717a74238c3b1ce1b"}
Sep 17 17:36:54.843037 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:54.842753    2574 scope.go:117] "RemoveContainer" containerID="ebf8d2f40362e72d769e0b7eb57645ce6f65cae04736275dc33e9f8d214f6a8c"
Sep 17 17:36:54.843037 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:54.842761    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f2e735a3-7d6c-48d9-84a6-40e168ba42ea","Type":"ContainerDied","Data":"47c3256f8159b4161eaf97423fb425cc73bd0e8d6a4cfdee1430036f912bd74c"}
Sep 17 17:36:54.843037 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:54.842785    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f2e735a3-7d6c-48d9-84a6-40e168ba42ea","Type":"ContainerDied","Data":"84249ab289434482a1981728db3dca7d87005c61ceaa6610dfdb27b3741e2001"}
Sep 17 17:36:54.843037 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:54.842809    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f2e735a3-7d6c-48d9-84a6-40e168ba42ea","Type":"ContainerDied","Data":"9976a3830c20ca5f8c2c4f399f6151f4875fd25d57ebf475e0e50a5ee24e7edb"}
Sep 17 17:36:54.850191 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:54.850174    2574 scope.go:117] "RemoveContainer" containerID="081b3c8c9915f35abc53be444108613dcdfd46bb4e1b5a318388f908b2273777"
Sep 17 17:36:54.858528 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:54.858506    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f2e735a3-7d6c-48d9-84a6-40e168ba42ea-configmap-metrics-client-ca\") pod \"f2e735a3-7d6c-48d9-84a6-40e168ba42ea\" (UID: \"f2e735a3-7d6c-48d9-84a6-40e168ba42ea\") "
Sep 17 17:36:54.858654 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:54.858541    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9lsp2\" (UniqueName: \"kubernetes.io/projected/f2e735a3-7d6c-48d9-84a6-40e168ba42ea-kube-api-access-9lsp2\") pod \"f2e735a3-7d6c-48d9-84a6-40e168ba42ea\" (UID: \"f2e735a3-7d6c-48d9-84a6-40e168ba42ea\") "
Sep 17 17:36:54.858654 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:54.858560    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f2e735a3-7d6c-48d9-84a6-40e168ba42ea-secret-grpc-tls\") pod \"f2e735a3-7d6c-48d9-84a6-40e168ba42ea\" (UID: \"f2e735a3-7d6c-48d9-84a6-40e168ba42ea\") "
Sep 17 17:36:54.858654 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:54.858577    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f2e735a3-7d6c-48d9-84a6-40e168ba42ea-web-config\") pod \"f2e735a3-7d6c-48d9-84a6-40e168ba42ea\" (UID: \"f2e735a3-7d6c-48d9-84a6-40e168ba42ea\") "
Sep 17 17:36:54.858654 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:54.858619    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f2e735a3-7d6c-48d9-84a6-40e168ba42ea-secret-kube-rbac-proxy\") pod \"f2e735a3-7d6c-48d9-84a6-40e168ba42ea\" (UID: \"f2e735a3-7d6c-48d9-84a6-40e168ba42ea\") "
Sep 17 17:36:54.858654 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:54.858645    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/f2e735a3-7d6c-48d9-84a6-40e168ba42ea-prometheus-k8s-db\") pod \"f2e735a3-7d6c-48d9-84a6-40e168ba42ea\" (UID: \"f2e735a3-7d6c-48d9-84a6-40e168ba42ea\") "
Sep 17 17:36:54.858901 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:54.858673    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/f2e735a3-7d6c-48d9-84a6-40e168ba42ea-prometheus-k8s-rulefiles-2\") pod \"f2e735a3-7d6c-48d9-84a6-40e168ba42ea\" (UID: \"f2e735a3-7d6c-48d9-84a6-40e168ba42ea\") "
Sep 17 17:36:54.858901 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:54.858704    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f2e735a3-7d6c-48d9-84a6-40e168ba42ea-prometheus-trusted-ca-bundle\") pod \"f2e735a3-7d6c-48d9-84a6-40e168ba42ea\" (UID: \"f2e735a3-7d6c-48d9-84a6-40e168ba42ea\") "
Sep 17 17:36:54.858901 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:54.858729    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/f2e735a3-7d6c-48d9-84a6-40e168ba42ea-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"f2e735a3-7d6c-48d9-84a6-40e168ba42ea\" (UID: \"f2e735a3-7d6c-48d9-84a6-40e168ba42ea\") "
Sep 17 17:36:54.858901 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:54.858760    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f2e735a3-7d6c-48d9-84a6-40e168ba42ea-configmap-kubelet-serving-ca-bundle\") pod \"f2e735a3-7d6c-48d9-84a6-40e168ba42ea\" (UID: \"f2e735a3-7d6c-48d9-84a6-40e168ba42ea\") "
Sep 17 17:36:54.858901 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:54.858782    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/f2e735a3-7d6c-48d9-84a6-40e168ba42ea-secret-metrics-client-certs\") pod \"f2e735a3-7d6c-48d9-84a6-40e168ba42ea\" (UID: \"f2e735a3-7d6c-48d9-84a6-40e168ba42ea\") "
Sep 17 17:36:54.858901 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:54.858817    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/f2e735a3-7d6c-48d9-84a6-40e168ba42ea-prometheus-k8s-rulefiles-0\") pod \"f2e735a3-7d6c-48d9-84a6-40e168ba42ea\" (UID: \"f2e735a3-7d6c-48d9-84a6-40e168ba42ea\") "
Sep 17 17:36:54.858901 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:54.858843    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/f2e735a3-7d6c-48d9-84a6-40e168ba42ea-secret-prometheus-k8s-tls\") pod \"f2e735a3-7d6c-48d9-84a6-40e168ba42ea\" (UID: \"f2e735a3-7d6c-48d9-84a6-40e168ba42ea\") "
Sep 17 17:36:54.858901 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:54.858869    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f2e735a3-7d6c-48d9-84a6-40e168ba42ea-configmap-serving-certs-ca-bundle\") pod \"f2e735a3-7d6c-48d9-84a6-40e168ba42ea\" (UID: \"f2e735a3-7d6c-48d9-84a6-40e168ba42ea\") "
Sep 17 17:36:54.858901 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:54.858896    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/f2e735a3-7d6c-48d9-84a6-40e168ba42ea-thanos-prometheus-http-client-file\") pod \"f2e735a3-7d6c-48d9-84a6-40e168ba42ea\" (UID: \"f2e735a3-7d6c-48d9-84a6-40e168ba42ea\") "
Sep 17 17:36:54.859384 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:54.858941    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f2e735a3-7d6c-48d9-84a6-40e168ba42ea-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"f2e735a3-7d6c-48d9-84a6-40e168ba42ea\" (UID: \"f2e735a3-7d6c-48d9-84a6-40e168ba42ea\") "
Sep 17 17:36:54.859384 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:54.858969    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/f2e735a3-7d6c-48d9-84a6-40e168ba42ea-prometheus-k8s-rulefiles-1\") pod \"f2e735a3-7d6c-48d9-84a6-40e168ba42ea\" (UID: \"f2e735a3-7d6c-48d9-84a6-40e168ba42ea\") "
Sep 17 17:36:54.859384 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:54.858998    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/f2e735a3-7d6c-48d9-84a6-40e168ba42ea-config\") pod \"f2e735a3-7d6c-48d9-84a6-40e168ba42ea\" (UID: \"f2e735a3-7d6c-48d9-84a6-40e168ba42ea\") "
Sep 17 17:36:54.859384 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:54.859020    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f2e735a3-7d6c-48d9-84a6-40e168ba42ea-config-out\") pod \"f2e735a3-7d6c-48d9-84a6-40e168ba42ea\" (UID: \"f2e735a3-7d6c-48d9-84a6-40e168ba42ea\") "
Sep 17 17:36:54.859384 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:54.859241    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2e735a3-7d6c-48d9-84a6-40e168ba42ea-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "f2e735a3-7d6c-48d9-84a6-40e168ba42ea" (UID: "f2e735a3-7d6c-48d9-84a6-40e168ba42ea"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:36:54.859384 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:54.859354    2574 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f2e735a3-7d6c-48d9-84a6-40e168ba42ea-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-134-96.ec2.internal\" DevicePath \"\""
Sep 17 17:36:54.859655 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:54.859638    2574 scope.go:117] "RemoveContainer" containerID="b48ba3b5af18adb7ebac1e07dd5923d84ca016bd2667598ac6cfc7de6e6849d4"
Sep 17 17:36:54.860306 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:54.860226    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2e735a3-7d6c-48d9-84a6-40e168ba42ea-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "f2e735a3-7d6c-48d9-84a6-40e168ba42ea" (UID: "f2e735a3-7d6c-48d9-84a6-40e168ba42ea"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:36:54.860306 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:54.860263    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2e735a3-7d6c-48d9-84a6-40e168ba42ea-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "f2e735a3-7d6c-48d9-84a6-40e168ba42ea" (UID: "f2e735a3-7d6c-48d9-84a6-40e168ba42ea"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:36:54.860514 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:54.860478    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2e735a3-7d6c-48d9-84a6-40e168ba42ea-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "f2e735a3-7d6c-48d9-84a6-40e168ba42ea" (UID: "f2e735a3-7d6c-48d9-84a6-40e168ba42ea"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:36:54.860628 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:54.860580    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2e735a3-7d6c-48d9-84a6-40e168ba42ea-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "f2e735a3-7d6c-48d9-84a6-40e168ba42ea" (UID: "f2e735a3-7d6c-48d9-84a6-40e168ba42ea"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:36:54.860738 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:54.860711    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2e735a3-7d6c-48d9-84a6-40e168ba42ea-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "f2e735a3-7d6c-48d9-84a6-40e168ba42ea" (UID: "f2e735a3-7d6c-48d9-84a6-40e168ba42ea"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:36:54.861661 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:54.861600    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2e735a3-7d6c-48d9-84a6-40e168ba42ea-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "f2e735a3-7d6c-48d9-84a6-40e168ba42ea" (UID: "f2e735a3-7d6c-48d9-84a6-40e168ba42ea"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:36:54.861661 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:54.861633    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2e735a3-7d6c-48d9-84a6-40e168ba42ea-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "f2e735a3-7d6c-48d9-84a6-40e168ba42ea" (UID: "f2e735a3-7d6c-48d9-84a6-40e168ba42ea"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:36:54.862893 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:54.862865    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f2e735a3-7d6c-48d9-84a6-40e168ba42ea-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "f2e735a3-7d6c-48d9-84a6-40e168ba42ea" (UID: "f2e735a3-7d6c-48d9-84a6-40e168ba42ea"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:36:54.863419 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:54.863385    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f2e735a3-7d6c-48d9-84a6-40e168ba42ea-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "f2e735a3-7d6c-48d9-84a6-40e168ba42ea" (UID: "f2e735a3-7d6c-48d9-84a6-40e168ba42ea"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:36:54.863567 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:54.863543    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f2e735a3-7d6c-48d9-84a6-40e168ba42ea-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "f2e735a3-7d6c-48d9-84a6-40e168ba42ea" (UID: "f2e735a3-7d6c-48d9-84a6-40e168ba42ea"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:36:54.863961 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:54.863942    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2e735a3-7d6c-48d9-84a6-40e168ba42ea-kube-api-access-9lsp2" (OuterVolumeSpecName: "kube-api-access-9lsp2") pod "f2e735a3-7d6c-48d9-84a6-40e168ba42ea" (UID: "f2e735a3-7d6c-48d9-84a6-40e168ba42ea"). InnerVolumeSpecName "kube-api-access-9lsp2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:36:54.864265 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:54.864229    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2e735a3-7d6c-48d9-84a6-40e168ba42ea-config-out" (OuterVolumeSpecName: "config-out") pod "f2e735a3-7d6c-48d9-84a6-40e168ba42ea" (UID: "f2e735a3-7d6c-48d9-84a6-40e168ba42ea"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:36:54.864504 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:54.864480    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f2e735a3-7d6c-48d9-84a6-40e168ba42ea-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "f2e735a3-7d6c-48d9-84a6-40e168ba42ea" (UID: "f2e735a3-7d6c-48d9-84a6-40e168ba42ea"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:36:54.864504 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:54.864489    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f2e735a3-7d6c-48d9-84a6-40e168ba42ea-config" (OuterVolumeSpecName: "config") pod "f2e735a3-7d6c-48d9-84a6-40e168ba42ea" (UID: "f2e735a3-7d6c-48d9-84a6-40e168ba42ea"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:36:54.864659 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:54.864585    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f2e735a3-7d6c-48d9-84a6-40e168ba42ea-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "f2e735a3-7d6c-48d9-84a6-40e168ba42ea" (UID: "f2e735a3-7d6c-48d9-84a6-40e168ba42ea"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:36:54.864735 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:54.864718    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f2e735a3-7d6c-48d9-84a6-40e168ba42ea-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "f2e735a3-7d6c-48d9-84a6-40e168ba42ea" (UID: "f2e735a3-7d6c-48d9-84a6-40e168ba42ea"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:36:54.865032 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:54.865006    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f2e735a3-7d6c-48d9-84a6-40e168ba42ea-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "f2e735a3-7d6c-48d9-84a6-40e168ba42ea" (UID: "f2e735a3-7d6c-48d9-84a6-40e168ba42ea"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:36:54.868667 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:54.868637    2574 scope.go:117] "RemoveContainer" containerID="d808f041a38de295686a2cd50b9d5e3ec2f7bb08b97e870717a74238c3b1ce1b"
Sep 17 17:36:54.873990 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:54.873971    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f2e735a3-7d6c-48d9-84a6-40e168ba42ea-web-config" (OuterVolumeSpecName: "web-config") pod "f2e735a3-7d6c-48d9-84a6-40e168ba42ea" (UID: "f2e735a3-7d6c-48d9-84a6-40e168ba42ea"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:36:54.875250 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:54.875234    2574 scope.go:117] "RemoveContainer" containerID="47c3256f8159b4161eaf97423fb425cc73bd0e8d6a4cfdee1430036f912bd74c"
Sep 17 17:36:54.879913 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:54.879893    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j4ds_d3f553e8-b416-4d7d-bbef-991a0404b287/konnectivity-agent/0.log" limit=1024
Sep 17 17:36:54.881382 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:54.881364    2574 scope.go:117] "RemoveContainer" containerID="84249ab289434482a1981728db3dca7d87005c61ceaa6610dfdb27b3741e2001"
Sep 17 17:36:54.887773 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:54.887755    2574 scope.go:117] "RemoveContainer" containerID="5bd1e9c20df384961796065ea0677bee6f4ed4c5932e6efbe47972a24370b8dd"
Sep 17 17:36:54.893554 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:54.893540    2574 scope.go:117] "RemoveContainer" containerID="ebf8d2f40362e72d769e0b7eb57645ce6f65cae04736275dc33e9f8d214f6a8c"
Sep 17 17:36:54.893803 ip-10-0-134-96 kubenswrapper[2574]: E0917 17:36:54.893784    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ebf8d2f40362e72d769e0b7eb57645ce6f65cae04736275dc33e9f8d214f6a8c\": container with ID starting with ebf8d2f40362e72d769e0b7eb57645ce6f65cae04736275dc33e9f8d214f6a8c not found: ID does not exist" containerID="ebf8d2f40362e72d769e0b7eb57645ce6f65cae04736275dc33e9f8d214f6a8c"
Sep 17 17:36:54.893859 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:54.893810    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ebf8d2f40362e72d769e0b7eb57645ce6f65cae04736275dc33e9f8d214f6a8c"} err="failed to get container status \"ebf8d2f40362e72d769e0b7eb57645ce6f65cae04736275dc33e9f8d214f6a8c\": rpc error: code = NotFound desc = could not find container \"ebf8d2f40362e72d769e0b7eb57645ce6f65cae04736275dc33e9f8d214f6a8c\": container with ID starting with ebf8d2f40362e72d769e0b7eb57645ce6f65cae04736275dc33e9f8d214f6a8c not found: ID does not exist"
Sep 17 17:36:54.893859 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:54.893826    2574 scope.go:117] "RemoveContainer" containerID="081b3c8c9915f35abc53be444108613dcdfd46bb4e1b5a318388f908b2273777"
Sep 17 17:36:54.894069 ip-10-0-134-96 kubenswrapper[2574]: E0917 17:36:54.894053    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"081b3c8c9915f35abc53be444108613dcdfd46bb4e1b5a318388f908b2273777\": container with ID starting with 081b3c8c9915f35abc53be444108613dcdfd46bb4e1b5a318388f908b2273777 not found: ID does not exist" containerID="081b3c8c9915f35abc53be444108613dcdfd46bb4e1b5a318388f908b2273777"
Sep 17 17:36:54.894105 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:54.894076    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"081b3c8c9915f35abc53be444108613dcdfd46bb4e1b5a318388f908b2273777"} err="failed to get container status \"081b3c8c9915f35abc53be444108613dcdfd46bb4e1b5a318388f908b2273777\": rpc error: code = NotFound desc = could not find container \"081b3c8c9915f35abc53be444108613dcdfd46bb4e1b5a318388f908b2273777\": container with ID starting with 081b3c8c9915f35abc53be444108613dcdfd46bb4e1b5a318388f908b2273777 not found: ID does not exist"
Sep 17 17:36:54.894105 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:54.894090    2574 scope.go:117] "RemoveContainer" containerID="b48ba3b5af18adb7ebac1e07dd5923d84ca016bd2667598ac6cfc7de6e6849d4"
Sep 17 17:36:54.894320 ip-10-0-134-96 kubenswrapper[2574]: E0917 17:36:54.894303    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b48ba3b5af18adb7ebac1e07dd5923d84ca016bd2667598ac6cfc7de6e6849d4\": container with ID starting with b48ba3b5af18adb7ebac1e07dd5923d84ca016bd2667598ac6cfc7de6e6849d4 not found: ID does not exist" containerID="b48ba3b5af18adb7ebac1e07dd5923d84ca016bd2667598ac6cfc7de6e6849d4"
Sep 17 17:36:54.894370 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:54.894324    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b48ba3b5af18adb7ebac1e07dd5923d84ca016bd2667598ac6cfc7de6e6849d4"} err="failed to get container status \"b48ba3b5af18adb7ebac1e07dd5923d84ca016bd2667598ac6cfc7de6e6849d4\": rpc error: code = NotFound desc = could not find container \"b48ba3b5af18adb7ebac1e07dd5923d84ca016bd2667598ac6cfc7de6e6849d4\": container with ID starting with b48ba3b5af18adb7ebac1e07dd5923d84ca016bd2667598ac6cfc7de6e6849d4 not found: ID does not exist"
Sep 17 17:36:54.894370 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:54.894337    2574 scope.go:117] "RemoveContainer" containerID="d808f041a38de295686a2cd50b9d5e3ec2f7bb08b97e870717a74238c3b1ce1b"
Sep 17 17:36:54.894567 ip-10-0-134-96 kubenswrapper[2574]: E0917 17:36:54.894552    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d808f041a38de295686a2cd50b9d5e3ec2f7bb08b97e870717a74238c3b1ce1b\": container with ID starting with d808f041a38de295686a2cd50b9d5e3ec2f7bb08b97e870717a74238c3b1ce1b not found: ID does not exist" containerID="d808f041a38de295686a2cd50b9d5e3ec2f7bb08b97e870717a74238c3b1ce1b"
Sep 17 17:36:54.894605 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:54.894573    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d808f041a38de295686a2cd50b9d5e3ec2f7bb08b97e870717a74238c3b1ce1b"} err="failed to get container status \"d808f041a38de295686a2cd50b9d5e3ec2f7bb08b97e870717a74238c3b1ce1b\": rpc error: code = NotFound desc = could not find container \"d808f041a38de295686a2cd50b9d5e3ec2f7bb08b97e870717a74238c3b1ce1b\": container with ID starting with d808f041a38de295686a2cd50b9d5e3ec2f7bb08b97e870717a74238c3b1ce1b not found: ID does not exist"
Sep 17 17:36:54.894605 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:54.894586    2574 scope.go:117] "RemoveContainer" containerID="47c3256f8159b4161eaf97423fb425cc73bd0e8d6a4cfdee1430036f912bd74c"
Sep 17 17:36:54.894796 ip-10-0-134-96 kubenswrapper[2574]: E0917 17:36:54.894782    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"47c3256f8159b4161eaf97423fb425cc73bd0e8d6a4cfdee1430036f912bd74c\": container with ID starting with 47c3256f8159b4161eaf97423fb425cc73bd0e8d6a4cfdee1430036f912bd74c not found: ID does not exist" containerID="47c3256f8159b4161eaf97423fb425cc73bd0e8d6a4cfdee1430036f912bd74c"
Sep 17 17:36:54.894830 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:54.894800    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47c3256f8159b4161eaf97423fb425cc73bd0e8d6a4cfdee1430036f912bd74c"} err="failed to get container status \"47c3256f8159b4161eaf97423fb425cc73bd0e8d6a4cfdee1430036f912bd74c\": rpc error: code = NotFound desc = could not find container \"47c3256f8159b4161eaf97423fb425cc73bd0e8d6a4cfdee1430036f912bd74c\": container with ID starting with 47c3256f8159b4161eaf97423fb425cc73bd0e8d6a4cfdee1430036f912bd74c not found: ID does not exist"
Sep 17 17:36:54.894830 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:54.894812    2574 scope.go:117] "RemoveContainer" containerID="84249ab289434482a1981728db3dca7d87005c61ceaa6610dfdb27b3741e2001"
Sep 17 17:36:54.895016 ip-10-0-134-96 kubenswrapper[2574]: E0917 17:36:54.895002    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"84249ab289434482a1981728db3dca7d87005c61ceaa6610dfdb27b3741e2001\": container with ID starting with 84249ab289434482a1981728db3dca7d87005c61ceaa6610dfdb27b3741e2001 not found: ID does not exist" containerID="84249ab289434482a1981728db3dca7d87005c61ceaa6610dfdb27b3741e2001"
Sep 17 17:36:54.895073 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:54.895021    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84249ab289434482a1981728db3dca7d87005c61ceaa6610dfdb27b3741e2001"} err="failed to get container status \"84249ab289434482a1981728db3dca7d87005c61ceaa6610dfdb27b3741e2001\": rpc error: code = NotFound desc = could not find container \"84249ab289434482a1981728db3dca7d87005c61ceaa6610dfdb27b3741e2001\": container with ID starting with 84249ab289434482a1981728db3dca7d87005c61ceaa6610dfdb27b3741e2001 not found: ID does not exist"
Sep 17 17:36:54.895073 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:54.895037    2574 scope.go:117] "RemoveContainer" containerID="5bd1e9c20df384961796065ea0677bee6f4ed4c5932e6efbe47972a24370b8dd"
Sep 17 17:36:54.895308 ip-10-0-134-96 kubenswrapper[2574]: E0917 17:36:54.895291    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5bd1e9c20df384961796065ea0677bee6f4ed4c5932e6efbe47972a24370b8dd\": container with ID starting with 5bd1e9c20df384961796065ea0677bee6f4ed4c5932e6efbe47972a24370b8dd not found: ID does not exist" containerID="5bd1e9c20df384961796065ea0677bee6f4ed4c5932e6efbe47972a24370b8dd"
Sep 17 17:36:54.895341 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:54.895313    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5bd1e9c20df384961796065ea0677bee6f4ed4c5932e6efbe47972a24370b8dd"} err="failed to get container status \"5bd1e9c20df384961796065ea0677bee6f4ed4c5932e6efbe47972a24370b8dd\": rpc error: code = NotFound desc = could not find container \"5bd1e9c20df384961796065ea0677bee6f4ed4c5932e6efbe47972a24370b8dd\": container with ID starting with 5bd1e9c20df384961796065ea0677bee6f4ed4c5932e6efbe47972a24370b8dd not found: ID does not exist"
Sep 17 17:36:54.895341 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:54.895325    2574 scope.go:117] "RemoveContainer" containerID="ebf8d2f40362e72d769e0b7eb57645ce6f65cae04736275dc33e9f8d214f6a8c"
Sep 17 17:36:54.895532 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:54.895515    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ebf8d2f40362e72d769e0b7eb57645ce6f65cae04736275dc33e9f8d214f6a8c"} err="failed to get container status \"ebf8d2f40362e72d769e0b7eb57645ce6f65cae04736275dc33e9f8d214f6a8c\": rpc error: code = NotFound desc = could not find container \"ebf8d2f40362e72d769e0b7eb57645ce6f65cae04736275dc33e9f8d214f6a8c\": container with ID starting with ebf8d2f40362e72d769e0b7eb57645ce6f65cae04736275dc33e9f8d214f6a8c not found: ID does not exist"
Sep 17 17:36:54.895573 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:54.895533    2574 scope.go:117] "RemoveContainer" containerID="081b3c8c9915f35abc53be444108613dcdfd46bb4e1b5a318388f908b2273777"
Sep 17 17:36:54.895724 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:54.895709    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"081b3c8c9915f35abc53be444108613dcdfd46bb4e1b5a318388f908b2273777"} err="failed to get container status \"081b3c8c9915f35abc53be444108613dcdfd46bb4e1b5a318388f908b2273777\": rpc error: code = NotFound desc = could not find container \"081b3c8c9915f35abc53be444108613dcdfd46bb4e1b5a318388f908b2273777\": container with ID starting with 081b3c8c9915f35abc53be444108613dcdfd46bb4e1b5a318388f908b2273777 not found: ID does not exist"
Sep 17 17:36:54.895761 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:54.895724    2574 scope.go:117] "RemoveContainer" containerID="b48ba3b5af18adb7ebac1e07dd5923d84ca016bd2667598ac6cfc7de6e6849d4"
Sep 17 17:36:54.895943 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:54.895926    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b48ba3b5af18adb7ebac1e07dd5923d84ca016bd2667598ac6cfc7de6e6849d4"} err="failed to get container status \"b48ba3b5af18adb7ebac1e07dd5923d84ca016bd2667598ac6cfc7de6e6849d4\": rpc error: code = NotFound desc = could not find container \"b48ba3b5af18adb7ebac1e07dd5923d84ca016bd2667598ac6cfc7de6e6849d4\": container with ID starting with b48ba3b5af18adb7ebac1e07dd5923d84ca016bd2667598ac6cfc7de6e6849d4 not found: ID does not exist"
Sep 17 17:36:54.895985 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:54.895943    2574 scope.go:117] "RemoveContainer" containerID="d808f041a38de295686a2cd50b9d5e3ec2f7bb08b97e870717a74238c3b1ce1b"
Sep 17 17:36:54.896176 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:54.896159    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d808f041a38de295686a2cd50b9d5e3ec2f7bb08b97e870717a74238c3b1ce1b"} err="failed to get container status \"d808f041a38de295686a2cd50b9d5e3ec2f7bb08b97e870717a74238c3b1ce1b\": rpc error: code = NotFound desc = could not find container \"d808f041a38de295686a2cd50b9d5e3ec2f7bb08b97e870717a74238c3b1ce1b\": container with ID starting with d808f041a38de295686a2cd50b9d5e3ec2f7bb08b97e870717a74238c3b1ce1b not found: ID does not exist"
Sep 17 17:36:54.896232 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:54.896177    2574 scope.go:117] "RemoveContainer" containerID="47c3256f8159b4161eaf97423fb425cc73bd0e8d6a4cfdee1430036f912bd74c"
Sep 17 17:36:54.896405 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:54.896391    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47c3256f8159b4161eaf97423fb425cc73bd0e8d6a4cfdee1430036f912bd74c"} err="failed to get container status \"47c3256f8159b4161eaf97423fb425cc73bd0e8d6a4cfdee1430036f912bd74c\": rpc error: code = NotFound desc = could not find container \"47c3256f8159b4161eaf97423fb425cc73bd0e8d6a4cfdee1430036f912bd74c\": container with ID starting with 47c3256f8159b4161eaf97423fb425cc73bd0e8d6a4cfdee1430036f912bd74c not found: ID does not exist"
Sep 17 17:36:54.896405 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:54.896404    2574 scope.go:117] "RemoveContainer" containerID="84249ab289434482a1981728db3dca7d87005c61ceaa6610dfdb27b3741e2001"
Sep 17 17:36:54.896597 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:54.896584    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84249ab289434482a1981728db3dca7d87005c61ceaa6610dfdb27b3741e2001"} err="failed to get container status \"84249ab289434482a1981728db3dca7d87005c61ceaa6610dfdb27b3741e2001\": rpc error: code = NotFound desc = could not find container \"84249ab289434482a1981728db3dca7d87005c61ceaa6610dfdb27b3741e2001\": container with ID starting with 84249ab289434482a1981728db3dca7d87005c61ceaa6610dfdb27b3741e2001 not found: ID does not exist"
Sep 17 17:36:54.896644 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:54.896597    2574 scope.go:117] "RemoveContainer" containerID="5bd1e9c20df384961796065ea0677bee6f4ed4c5932e6efbe47972a24370b8dd"
Sep 17 17:36:54.896763 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:54.896751    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5bd1e9c20df384961796065ea0677bee6f4ed4c5932e6efbe47972a24370b8dd"} err="failed to get container status \"5bd1e9c20df384961796065ea0677bee6f4ed4c5932e6efbe47972a24370b8dd\": rpc error: code = NotFound desc = could not find container \"5bd1e9c20df384961796065ea0677bee6f4ed4c5932e6efbe47972a24370b8dd\": container with ID starting with 5bd1e9c20df384961796065ea0677bee6f4ed4c5932e6efbe47972a24370b8dd not found: ID does not exist"
Sep 17 17:36:54.896763 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:54.896763    2574 scope.go:117] "RemoveContainer" containerID="ebf8d2f40362e72d769e0b7eb57645ce6f65cae04736275dc33e9f8d214f6a8c"
Sep 17 17:36:54.896968 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:54.896952    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ebf8d2f40362e72d769e0b7eb57645ce6f65cae04736275dc33e9f8d214f6a8c"} err="failed to get container status \"ebf8d2f40362e72d769e0b7eb57645ce6f65cae04736275dc33e9f8d214f6a8c\": rpc error: code = NotFound desc = could not find container \"ebf8d2f40362e72d769e0b7eb57645ce6f65cae04736275dc33e9f8d214f6a8c\": container with ID starting with ebf8d2f40362e72d769e0b7eb57645ce6f65cae04736275dc33e9f8d214f6a8c not found: ID does not exist"
Sep 17 17:36:54.897016 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:54.896968    2574 scope.go:117] "RemoveContainer" containerID="081b3c8c9915f35abc53be444108613dcdfd46bb4e1b5a318388f908b2273777"
Sep 17 17:36:54.897193 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:54.897164    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"081b3c8c9915f35abc53be444108613dcdfd46bb4e1b5a318388f908b2273777"} err="failed to get container status \"081b3c8c9915f35abc53be444108613dcdfd46bb4e1b5a318388f908b2273777\": rpc error: code = NotFound desc = could not find container \"081b3c8c9915f35abc53be444108613dcdfd46bb4e1b5a318388f908b2273777\": container with ID starting with 081b3c8c9915f35abc53be444108613dcdfd46bb4e1b5a318388f908b2273777 not found: ID does not exist"
Sep 17 17:36:54.897193 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:54.897185    2574 scope.go:117] "RemoveContainer" containerID="b48ba3b5af18adb7ebac1e07dd5923d84ca016bd2667598ac6cfc7de6e6849d4"
Sep 17 17:36:54.897409 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:54.897393    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b48ba3b5af18adb7ebac1e07dd5923d84ca016bd2667598ac6cfc7de6e6849d4"} err="failed to get container status \"b48ba3b5af18adb7ebac1e07dd5923d84ca016bd2667598ac6cfc7de6e6849d4\": rpc error: code = NotFound desc = could not find container \"b48ba3b5af18adb7ebac1e07dd5923d84ca016bd2667598ac6cfc7de6e6849d4\": container with ID starting with b48ba3b5af18adb7ebac1e07dd5923d84ca016bd2667598ac6cfc7de6e6849d4 not found: ID does not exist"
Sep 17 17:36:54.897409 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:54.897409    2574 scope.go:117] "RemoveContainer" containerID="d808f041a38de295686a2cd50b9d5e3ec2f7bb08b97e870717a74238c3b1ce1b"
Sep 17 17:36:54.897604 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:54.897587    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d808f041a38de295686a2cd50b9d5e3ec2f7bb08b97e870717a74238c3b1ce1b"} err="failed to get container status \"d808f041a38de295686a2cd50b9d5e3ec2f7bb08b97e870717a74238c3b1ce1b\": rpc error: code = NotFound desc = could not find container \"d808f041a38de295686a2cd50b9d5e3ec2f7bb08b97e870717a74238c3b1ce1b\": container with ID starting with d808f041a38de295686a2cd50b9d5e3ec2f7bb08b97e870717a74238c3b1ce1b not found: ID does not exist"
Sep 17 17:36:54.897667 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:54.897606    2574 scope.go:117] "RemoveContainer" containerID="47c3256f8159b4161eaf97423fb425cc73bd0e8d6a4cfdee1430036f912bd74c"
Sep 17 17:36:54.897812 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:54.897794    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47c3256f8159b4161eaf97423fb425cc73bd0e8d6a4cfdee1430036f912bd74c"} err="failed to get container status \"47c3256f8159b4161eaf97423fb425cc73bd0e8d6a4cfdee1430036f912bd74c\": rpc error: code = NotFound desc = could not find container \"47c3256f8159b4161eaf97423fb425cc73bd0e8d6a4cfdee1430036f912bd74c\": container with ID starting with 47c3256f8159b4161eaf97423fb425cc73bd0e8d6a4cfdee1430036f912bd74c not found: ID does not exist"
Sep 17 17:36:54.897859 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:54.897814    2574 scope.go:117] "RemoveContainer" containerID="84249ab289434482a1981728db3dca7d87005c61ceaa6610dfdb27b3741e2001"
Sep 17 17:36:54.897999 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:54.897985    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84249ab289434482a1981728db3dca7d87005c61ceaa6610dfdb27b3741e2001"} err="failed to get container status \"84249ab289434482a1981728db3dca7d87005c61ceaa6610dfdb27b3741e2001\": rpc error: code = NotFound desc = could not find container \"84249ab289434482a1981728db3dca7d87005c61ceaa6610dfdb27b3741e2001\": container with ID starting with 84249ab289434482a1981728db3dca7d87005c61ceaa6610dfdb27b3741e2001 not found: ID does not exist"
Sep 17 17:36:54.898080 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:54.897998    2574 scope.go:117] "RemoveContainer" containerID="5bd1e9c20df384961796065ea0677bee6f4ed4c5932e6efbe47972a24370b8dd"
Sep 17 17:36:54.898191 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:54.898173    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5bd1e9c20df384961796065ea0677bee6f4ed4c5932e6efbe47972a24370b8dd"} err="failed to get container status \"5bd1e9c20df384961796065ea0677bee6f4ed4c5932e6efbe47972a24370b8dd\": rpc error: code = NotFound desc = could not find container \"5bd1e9c20df384961796065ea0677bee6f4ed4c5932e6efbe47972a24370b8dd\": container with ID starting with 5bd1e9c20df384961796065ea0677bee6f4ed4c5932e6efbe47972a24370b8dd not found: ID does not exist"
Sep 17 17:36:54.898248 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:54.898192    2574 scope.go:117] "RemoveContainer" containerID="ebf8d2f40362e72d769e0b7eb57645ce6f65cae04736275dc33e9f8d214f6a8c"
Sep 17 17:36:54.898393 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:54.898377    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ebf8d2f40362e72d769e0b7eb57645ce6f65cae04736275dc33e9f8d214f6a8c"} err="failed to get container status \"ebf8d2f40362e72d769e0b7eb57645ce6f65cae04736275dc33e9f8d214f6a8c\": rpc error: code = NotFound desc = could not find container \"ebf8d2f40362e72d769e0b7eb57645ce6f65cae04736275dc33e9f8d214f6a8c\": container with ID starting with ebf8d2f40362e72d769e0b7eb57645ce6f65cae04736275dc33e9f8d214f6a8c not found: ID does not exist"
Sep 17 17:36:54.898445 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:54.898393    2574 scope.go:117] "RemoveContainer" containerID="081b3c8c9915f35abc53be444108613dcdfd46bb4e1b5a318388f908b2273777"
Sep 17 17:36:54.898580 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:54.898564    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"081b3c8c9915f35abc53be444108613dcdfd46bb4e1b5a318388f908b2273777"} err="failed to get container status \"081b3c8c9915f35abc53be444108613dcdfd46bb4e1b5a318388f908b2273777\": rpc error: code = NotFound desc = could not find container \"081b3c8c9915f35abc53be444108613dcdfd46bb4e1b5a318388f908b2273777\": container with ID starting with 081b3c8c9915f35abc53be444108613dcdfd46bb4e1b5a318388f908b2273777 not found: ID does not exist"
Sep 17 17:36:54.898631 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:54.898581    2574 scope.go:117] "RemoveContainer" containerID="b48ba3b5af18adb7ebac1e07dd5923d84ca016bd2667598ac6cfc7de6e6849d4"
Sep 17 17:36:54.898779 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:54.898764    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b48ba3b5af18adb7ebac1e07dd5923d84ca016bd2667598ac6cfc7de6e6849d4"} err="failed to get container status \"b48ba3b5af18adb7ebac1e07dd5923d84ca016bd2667598ac6cfc7de6e6849d4\": rpc error: code = NotFound desc = could not find container \"b48ba3b5af18adb7ebac1e07dd5923d84ca016bd2667598ac6cfc7de6e6849d4\": container with ID starting with b48ba3b5af18adb7ebac1e07dd5923d84ca016bd2667598ac6cfc7de6e6849d4 not found: ID does not exist"
Sep 17 17:36:54.898817 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:54.898780    2574 scope.go:117] "RemoveContainer" containerID="d808f041a38de295686a2cd50b9d5e3ec2f7bb08b97e870717a74238c3b1ce1b"
Sep 17 17:36:54.898984 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:54.898969    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d808f041a38de295686a2cd50b9d5e3ec2f7bb08b97e870717a74238c3b1ce1b"} err="failed to get container status \"d808f041a38de295686a2cd50b9d5e3ec2f7bb08b97e870717a74238c3b1ce1b\": rpc error: code = NotFound desc = could not find container \"d808f041a38de295686a2cd50b9d5e3ec2f7bb08b97e870717a74238c3b1ce1b\": container with ID starting with d808f041a38de295686a2cd50b9d5e3ec2f7bb08b97e870717a74238c3b1ce1b not found: ID does not exist"
Sep 17 17:36:54.899025 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:54.898985    2574 scope.go:117] "RemoveContainer" containerID="47c3256f8159b4161eaf97423fb425cc73bd0e8d6a4cfdee1430036f912bd74c"
Sep 17 17:36:54.899202 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:54.899183    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47c3256f8159b4161eaf97423fb425cc73bd0e8d6a4cfdee1430036f912bd74c"} err="failed to get container status \"47c3256f8159b4161eaf97423fb425cc73bd0e8d6a4cfdee1430036f912bd74c\": rpc error: code = NotFound desc = could not find container \"47c3256f8159b4161eaf97423fb425cc73bd0e8d6a4cfdee1430036f912bd74c\": container with ID starting with 47c3256f8159b4161eaf97423fb425cc73bd0e8d6a4cfdee1430036f912bd74c not found: ID does not exist"
Sep 17 17:36:54.899245 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:54.899202    2574 scope.go:117] "RemoveContainer" containerID="84249ab289434482a1981728db3dca7d87005c61ceaa6610dfdb27b3741e2001"
Sep 17 17:36:54.899393 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:54.899377    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84249ab289434482a1981728db3dca7d87005c61ceaa6610dfdb27b3741e2001"} err="failed to get container status \"84249ab289434482a1981728db3dca7d87005c61ceaa6610dfdb27b3741e2001\": rpc error: code = NotFound desc = could not find container \"84249ab289434482a1981728db3dca7d87005c61ceaa6610dfdb27b3741e2001\": container with ID starting with 84249ab289434482a1981728db3dca7d87005c61ceaa6610dfdb27b3741e2001 not found: ID does not exist"
Sep 17 17:36:54.899426 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:54.899394    2574 scope.go:117] "RemoveContainer" containerID="5bd1e9c20df384961796065ea0677bee6f4ed4c5932e6efbe47972a24370b8dd"
Sep 17 17:36:54.899586 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:54.899565    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5bd1e9c20df384961796065ea0677bee6f4ed4c5932e6efbe47972a24370b8dd"} err="failed to get container status \"5bd1e9c20df384961796065ea0677bee6f4ed4c5932e6efbe47972a24370b8dd\": rpc error: code = NotFound desc = could not find container \"5bd1e9c20df384961796065ea0677bee6f4ed4c5932e6efbe47972a24370b8dd\": container with ID starting with 5bd1e9c20df384961796065ea0677bee6f4ed4c5932e6efbe47972a24370b8dd not found: ID does not exist"
Sep 17 17:36:54.899630 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:54.899586    2574 scope.go:117] "RemoveContainer" containerID="ebf8d2f40362e72d769e0b7eb57645ce6f65cae04736275dc33e9f8d214f6a8c"
Sep 17 17:36:54.899792 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:54.899773    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ebf8d2f40362e72d769e0b7eb57645ce6f65cae04736275dc33e9f8d214f6a8c"} err="failed to get container status \"ebf8d2f40362e72d769e0b7eb57645ce6f65cae04736275dc33e9f8d214f6a8c\": rpc error: code = NotFound desc = could not find container \"ebf8d2f40362e72d769e0b7eb57645ce6f65cae04736275dc33e9f8d214f6a8c\": container with ID starting with ebf8d2f40362e72d769e0b7eb57645ce6f65cae04736275dc33e9f8d214f6a8c not found: ID does not exist"
Sep 17 17:36:54.899829 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:54.899793    2574 scope.go:117] "RemoveContainer" containerID="081b3c8c9915f35abc53be444108613dcdfd46bb4e1b5a318388f908b2273777"
Sep 17 17:36:54.900011 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:54.899993    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"081b3c8c9915f35abc53be444108613dcdfd46bb4e1b5a318388f908b2273777"} err="failed to get container status \"081b3c8c9915f35abc53be444108613dcdfd46bb4e1b5a318388f908b2273777\": rpc error: code = NotFound desc = could not find container \"081b3c8c9915f35abc53be444108613dcdfd46bb4e1b5a318388f908b2273777\": container with ID starting with 081b3c8c9915f35abc53be444108613dcdfd46bb4e1b5a318388f908b2273777 not found: ID does not exist"
Sep 17 17:36:54.900080 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:54.900010    2574 scope.go:117] "RemoveContainer" containerID="b48ba3b5af18adb7ebac1e07dd5923d84ca016bd2667598ac6cfc7de6e6849d4"
Sep 17 17:36:54.900240 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:54.900222    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b48ba3b5af18adb7ebac1e07dd5923d84ca016bd2667598ac6cfc7de6e6849d4"} err="failed to get container status \"b48ba3b5af18adb7ebac1e07dd5923d84ca016bd2667598ac6cfc7de6e6849d4\": rpc error: code = NotFound desc = could not find container \"b48ba3b5af18adb7ebac1e07dd5923d84ca016bd2667598ac6cfc7de6e6849d4\": container with ID starting with b48ba3b5af18adb7ebac1e07dd5923d84ca016bd2667598ac6cfc7de6e6849d4 not found: ID does not exist"
Sep 17 17:36:54.900283 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:54.900240    2574 scope.go:117] "RemoveContainer" containerID="d808f041a38de295686a2cd50b9d5e3ec2f7bb08b97e870717a74238c3b1ce1b"
Sep 17 17:36:54.900451 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:54.900433    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d808f041a38de295686a2cd50b9d5e3ec2f7bb08b97e870717a74238c3b1ce1b"} err="failed to get container status \"d808f041a38de295686a2cd50b9d5e3ec2f7bb08b97e870717a74238c3b1ce1b\": rpc error: code = NotFound desc = could not find container \"d808f041a38de295686a2cd50b9d5e3ec2f7bb08b97e870717a74238c3b1ce1b\": container with ID starting with d808f041a38de295686a2cd50b9d5e3ec2f7bb08b97e870717a74238c3b1ce1b not found: ID does not exist"
Sep 17 17:36:54.900514 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:54.900453    2574 scope.go:117] "RemoveContainer" containerID="47c3256f8159b4161eaf97423fb425cc73bd0e8d6a4cfdee1430036f912bd74c"
Sep 17 17:36:54.900702 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:54.900683    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47c3256f8159b4161eaf97423fb425cc73bd0e8d6a4cfdee1430036f912bd74c"} err="failed to get container status \"47c3256f8159b4161eaf97423fb425cc73bd0e8d6a4cfdee1430036f912bd74c\": rpc error: code = NotFound desc = could not find container \"47c3256f8159b4161eaf97423fb425cc73bd0e8d6a4cfdee1430036f912bd74c\": container with ID starting with 47c3256f8159b4161eaf97423fb425cc73bd0e8d6a4cfdee1430036f912bd74c not found: ID does not exist"
Sep 17 17:36:54.900747 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:54.900702    2574 scope.go:117] "RemoveContainer" containerID="84249ab289434482a1981728db3dca7d87005c61ceaa6610dfdb27b3741e2001"
Sep 17 17:36:54.900903 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:54.900886    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84249ab289434482a1981728db3dca7d87005c61ceaa6610dfdb27b3741e2001"} err="failed to get container status \"84249ab289434482a1981728db3dca7d87005c61ceaa6610dfdb27b3741e2001\": rpc error: code = NotFound desc = could not find container \"84249ab289434482a1981728db3dca7d87005c61ceaa6610dfdb27b3741e2001\": container with ID starting with 84249ab289434482a1981728db3dca7d87005c61ceaa6610dfdb27b3741e2001 not found: ID does not exist"
Sep 17 17:36:54.900960 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:54.900904    2574 scope.go:117] "RemoveContainer" containerID="5bd1e9c20df384961796065ea0677bee6f4ed4c5932e6efbe47972a24370b8dd"
Sep 17 17:36:54.901183 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:54.901161    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5bd1e9c20df384961796065ea0677bee6f4ed4c5932e6efbe47972a24370b8dd"} err="failed to get container status \"5bd1e9c20df384961796065ea0677bee6f4ed4c5932e6efbe47972a24370b8dd\": rpc error: code = NotFound desc = could not find container \"5bd1e9c20df384961796065ea0677bee6f4ed4c5932e6efbe47972a24370b8dd\": container with ID starting with 5bd1e9c20df384961796065ea0677bee6f4ed4c5932e6efbe47972a24370b8dd not found: ID does not exist"
Sep 17 17:36:54.901260 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:54.901185    2574 scope.go:117] "RemoveContainer" containerID="ebf8d2f40362e72d769e0b7eb57645ce6f65cae04736275dc33e9f8d214f6a8c"
Sep 17 17:36:54.901417 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:54.901401    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ebf8d2f40362e72d769e0b7eb57645ce6f65cae04736275dc33e9f8d214f6a8c"} err="failed to get container status \"ebf8d2f40362e72d769e0b7eb57645ce6f65cae04736275dc33e9f8d214f6a8c\": rpc error: code = NotFound desc = could not find container \"ebf8d2f40362e72d769e0b7eb57645ce6f65cae04736275dc33e9f8d214f6a8c\": container with ID starting with ebf8d2f40362e72d769e0b7eb57645ce6f65cae04736275dc33e9f8d214f6a8c not found: ID does not exist"
Sep 17 17:36:54.901473 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:54.901417    2574 scope.go:117] "RemoveContainer" containerID="081b3c8c9915f35abc53be444108613dcdfd46bb4e1b5a318388f908b2273777"
Sep 17 17:36:54.901608 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:54.901586    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"081b3c8c9915f35abc53be444108613dcdfd46bb4e1b5a318388f908b2273777"} err="failed to get container status \"081b3c8c9915f35abc53be444108613dcdfd46bb4e1b5a318388f908b2273777\": rpc error: code = NotFound desc = could not find container \"081b3c8c9915f35abc53be444108613dcdfd46bb4e1b5a318388f908b2273777\": container with ID starting with 081b3c8c9915f35abc53be444108613dcdfd46bb4e1b5a318388f908b2273777 not found: ID does not exist"
Sep 17 17:36:54.901647 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:54.901610    2574 scope.go:117] "RemoveContainer" containerID="b48ba3b5af18adb7ebac1e07dd5923d84ca016bd2667598ac6cfc7de6e6849d4"
Sep 17 17:36:54.901799 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:54.901783    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b48ba3b5af18adb7ebac1e07dd5923d84ca016bd2667598ac6cfc7de6e6849d4"} err="failed to get container status \"b48ba3b5af18adb7ebac1e07dd5923d84ca016bd2667598ac6cfc7de6e6849d4\": rpc error: code = NotFound desc = could not find container \"b48ba3b5af18adb7ebac1e07dd5923d84ca016bd2667598ac6cfc7de6e6849d4\": container with ID starting with b48ba3b5af18adb7ebac1e07dd5923d84ca016bd2667598ac6cfc7de6e6849d4 not found: ID does not exist"
Sep 17 17:36:54.901861 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:54.901801    2574 scope.go:117] "RemoveContainer" containerID="d808f041a38de295686a2cd50b9d5e3ec2f7bb08b97e870717a74238c3b1ce1b"
Sep 17 17:36:54.902033 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:54.902018    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d808f041a38de295686a2cd50b9d5e3ec2f7bb08b97e870717a74238c3b1ce1b"} err="failed to get container status \"d808f041a38de295686a2cd50b9d5e3ec2f7bb08b97e870717a74238c3b1ce1b\": rpc error: code = NotFound desc = could not find container \"d808f041a38de295686a2cd50b9d5e3ec2f7bb08b97e870717a74238c3b1ce1b\": container with ID starting with d808f041a38de295686a2cd50b9d5e3ec2f7bb08b97e870717a74238c3b1ce1b not found: ID does not exist"
Sep 17 17:36:54.902095 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:54.902033    2574 scope.go:117] "RemoveContainer" containerID="47c3256f8159b4161eaf97423fb425cc73bd0e8d6a4cfdee1430036f912bd74c"
Sep 17 17:36:54.902279 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:54.902263    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47c3256f8159b4161eaf97423fb425cc73bd0e8d6a4cfdee1430036f912bd74c"} err="failed to get container status \"47c3256f8159b4161eaf97423fb425cc73bd0e8d6a4cfdee1430036f912bd74c\": rpc error: code = NotFound desc = could not find container \"47c3256f8159b4161eaf97423fb425cc73bd0e8d6a4cfdee1430036f912bd74c\": container with ID starting with 47c3256f8159b4161eaf97423fb425cc73bd0e8d6a4cfdee1430036f912bd74c not found: ID does not exist"
Sep 17 17:36:54.902342 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:54.902281    2574 scope.go:117] "RemoveContainer" containerID="84249ab289434482a1981728db3dca7d87005c61ceaa6610dfdb27b3741e2001"
Sep 17 17:36:54.902483 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:54.902463    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84249ab289434482a1981728db3dca7d87005c61ceaa6610dfdb27b3741e2001"} err="failed to get container status \"84249ab289434482a1981728db3dca7d87005c61ceaa6610dfdb27b3741e2001\": rpc error: code = NotFound desc = could not find container \"84249ab289434482a1981728db3dca7d87005c61ceaa6610dfdb27b3741e2001\": container with ID starting with 84249ab289434482a1981728db3dca7d87005c61ceaa6610dfdb27b3741e2001 not found: ID does not exist"
Sep 17 17:36:54.902526 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:54.902484    2574 scope.go:117] "RemoveContainer" containerID="5bd1e9c20df384961796065ea0677bee6f4ed4c5932e6efbe47972a24370b8dd"
Sep 17 17:36:54.902677 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:54.902660    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5bd1e9c20df384961796065ea0677bee6f4ed4c5932e6efbe47972a24370b8dd"} err="failed to get container status \"5bd1e9c20df384961796065ea0677bee6f4ed4c5932e6efbe47972a24370b8dd\": rpc error: code = NotFound desc = could not find container \"5bd1e9c20df384961796065ea0677bee6f4ed4c5932e6efbe47972a24370b8dd\": container with ID starting with 5bd1e9c20df384961796065ea0677bee6f4ed4c5932e6efbe47972a24370b8dd not found: ID does not exist"
Sep 17 17:36:54.951147 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:54.951128    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j4ds_d3f553e8-b416-4d7d-bbef-991a0404b287/konnectivity-agent/0.log" limit=1024
Sep 17 17:36:54.960266 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:54.960249    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f2e735a3-7d6c-48d9-84a6-40e168ba42ea-tls-assets\") pod \"f2e735a3-7d6c-48d9-84a6-40e168ba42ea\" (UID: \"f2e735a3-7d6c-48d9-84a6-40e168ba42ea\") "
Sep 17 17:36:54.960372 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:54.960359    2574 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f2e735a3-7d6c-48d9-84a6-40e168ba42ea-secret-kube-rbac-proxy\") on node \"ip-10-0-134-96.ec2.internal\" DevicePath \"\""
Sep 17 17:36:54.960435 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:54.960378    2574 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/f2e735a3-7d6c-48d9-84a6-40e168ba42ea-prometheus-k8s-db\") on node \"ip-10-0-134-96.ec2.internal\" DevicePath \"\""
Sep 17 17:36:54.960435 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:54.960390    2574 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/f2e735a3-7d6c-48d9-84a6-40e168ba42ea-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-134-96.ec2.internal\" DevicePath \"\""
Sep 17 17:36:54.960435 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:54.960399    2574 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f2e735a3-7d6c-48d9-84a6-40e168ba42ea-prometheus-trusted-ca-bundle\") on node \"ip-10-0-134-96.ec2.internal\" DevicePath \"\""
Sep 17 17:36:54.960435 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:54.960409    2574 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/f2e735a3-7d6c-48d9-84a6-40e168ba42ea-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-134-96.ec2.internal\" DevicePath \"\""
Sep 17 17:36:54.960435 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:54.960418    2574 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/f2e735a3-7d6c-48d9-84a6-40e168ba42ea-secret-metrics-client-certs\") on node \"ip-10-0-134-96.ec2.internal\" DevicePath \"\""
Sep 17 17:36:54.960435 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:54.960426    2574 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/f2e735a3-7d6c-48d9-84a6-40e168ba42ea-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-134-96.ec2.internal\" DevicePath \"\""
Sep 17 17:36:54.960435 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:54.960434    2574 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/f2e735a3-7d6c-48d9-84a6-40e168ba42ea-secret-prometheus-k8s-tls\") on node \"ip-10-0-134-96.ec2.internal\" DevicePath \"\""
Sep 17 17:36:54.960726 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:54.960442    2574 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f2e735a3-7d6c-48d9-84a6-40e168ba42ea-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-134-96.ec2.internal\" DevicePath \"\""
Sep 17 17:36:54.960726 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:54.960452    2574 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/f2e735a3-7d6c-48d9-84a6-40e168ba42ea-thanos-prometheus-http-client-file\") on node \"ip-10-0-134-96.ec2.internal\" DevicePath \"\""
Sep 17 17:36:54.960726 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:54.960465    2574 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f2e735a3-7d6c-48d9-84a6-40e168ba42ea-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-134-96.ec2.internal\" DevicePath \"\""
Sep 17 17:36:54.960726 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:54.960473    2574 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/f2e735a3-7d6c-48d9-84a6-40e168ba42ea-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-134-96.ec2.internal\" DevicePath \"\""
Sep 17 17:36:54.960726 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:54.960481    2574 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/f2e735a3-7d6c-48d9-84a6-40e168ba42ea-config\") on node \"ip-10-0-134-96.ec2.internal\" DevicePath \"\""
Sep 17 17:36:54.960726 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:54.960488    2574 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f2e735a3-7d6c-48d9-84a6-40e168ba42ea-config-out\") on node \"ip-10-0-134-96.ec2.internal\" DevicePath \"\""
Sep 17 17:36:54.960726 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:54.960496    2574 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f2e735a3-7d6c-48d9-84a6-40e168ba42ea-configmap-metrics-client-ca\") on node \"ip-10-0-134-96.ec2.internal\" DevicePath \"\""
Sep 17 17:36:54.960726 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:54.960503    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9lsp2\" (UniqueName: \"kubernetes.io/projected/f2e735a3-7d6c-48d9-84a6-40e168ba42ea-kube-api-access-9lsp2\") on node \"ip-10-0-134-96.ec2.internal\" DevicePath \"\""
Sep 17 17:36:54.960726 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:54.960511    2574 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f2e735a3-7d6c-48d9-84a6-40e168ba42ea-secret-grpc-tls\") on node \"ip-10-0-134-96.ec2.internal\" DevicePath \"\""
Sep 17 17:36:54.960726 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:54.960518    2574 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f2e735a3-7d6c-48d9-84a6-40e168ba42ea-web-config\") on node \"ip-10-0-134-96.ec2.internal\" DevicePath \"\""
Sep 17 17:36:54.961998 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:54.961979    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2e735a3-7d6c-48d9-84a6-40e168ba42ea-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "f2e735a3-7d6c-48d9-84a6-40e168ba42ea" (UID: "f2e735a3-7d6c-48d9-84a6-40e168ba42ea"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:36:55.061483 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:55.061458    2574 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f2e735a3-7d6c-48d9-84a6-40e168ba42ea-tls-assets\") on node \"ip-10-0-134-96.ec2.internal\" DevicePath \"\""
Sep 17 17:36:55.169389 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:55.169364    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 17 17:36:55.182924 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:55.182900    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 17 17:36:55.211361 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:55.211340    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 17 17:36:55.211613 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:55.211599    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f2e735a3-7d6c-48d9-84a6-40e168ba42ea" containerName="kube-rbac-proxy"
Sep 17 17:36:55.211649 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:55.211617    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="f2e735a3-7d6c-48d9-84a6-40e168ba42ea" containerName="kube-rbac-proxy"
Sep 17 17:36:55.211649 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:55.211626    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f2e735a3-7d6c-48d9-84a6-40e168ba42ea" containerName="prometheus"
Sep 17 17:36:55.211649 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:55.211631    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="f2e735a3-7d6c-48d9-84a6-40e168ba42ea" containerName="prometheus"
Sep 17 17:36:55.211649 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:55.211639    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="035b396a-6ac7-4fe0-a639-104d2ff1bad3" containerName="console"
Sep 17 17:36:55.211649 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:55.211644    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="035b396a-6ac7-4fe0-a639-104d2ff1bad3" containerName="console"
Sep 17 17:36:55.211811 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:55.211652    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f2e735a3-7d6c-48d9-84a6-40e168ba42ea" containerName="init-config-reloader"
Sep 17 17:36:55.211811 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:55.211658    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="f2e735a3-7d6c-48d9-84a6-40e168ba42ea" containerName="init-config-reloader"
Sep 17 17:36:55.211811 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:55.211663    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f2e735a3-7d6c-48d9-84a6-40e168ba42ea" containerName="thanos-sidecar"
Sep 17 17:36:55.211811 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:55.211668    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="f2e735a3-7d6c-48d9-84a6-40e168ba42ea" containerName="thanos-sidecar"
Sep 17 17:36:55.211811 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:55.211677    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f2e735a3-7d6c-48d9-84a6-40e168ba42ea" containerName="kube-rbac-proxy-web"
Sep 17 17:36:55.211811 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:55.211685    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="f2e735a3-7d6c-48d9-84a6-40e168ba42ea" containerName="kube-rbac-proxy-web"
Sep 17 17:36:55.211811 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:55.211693    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f2e735a3-7d6c-48d9-84a6-40e168ba42ea" containerName="kube-rbac-proxy-thanos"
Sep 17 17:36:55.211811 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:55.211698    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="f2e735a3-7d6c-48d9-84a6-40e168ba42ea" containerName="kube-rbac-proxy-thanos"
Sep 17 17:36:55.211811 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:55.211707    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f2e735a3-7d6c-48d9-84a6-40e168ba42ea" containerName="config-reloader"
Sep 17 17:36:55.211811 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:55.211712    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="f2e735a3-7d6c-48d9-84a6-40e168ba42ea" containerName="config-reloader"
Sep 17 17:36:55.211811 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:55.211757    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="f2e735a3-7d6c-48d9-84a6-40e168ba42ea" containerName="kube-rbac-proxy-thanos"
Sep 17 17:36:55.211811 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:55.211769    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="035b396a-6ac7-4fe0-a639-104d2ff1bad3" containerName="console"
Sep 17 17:36:55.211811 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:55.211776    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="f2e735a3-7d6c-48d9-84a6-40e168ba42ea" containerName="kube-rbac-proxy-web"
Sep 17 17:36:55.211811 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:55.211782    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="f2e735a3-7d6c-48d9-84a6-40e168ba42ea" containerName="config-reloader"
Sep 17 17:36:55.211811 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:55.211787    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="f2e735a3-7d6c-48d9-84a6-40e168ba42ea" containerName="prometheus"
Sep 17 17:36:55.211811 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:55.211792    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="f2e735a3-7d6c-48d9-84a6-40e168ba42ea" containerName="thanos-sidecar"
Sep 17 17:36:55.211811 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:55.211797    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="f2e735a3-7d6c-48d9-84a6-40e168ba42ea" containerName="kube-rbac-proxy"
Sep 17 17:36:55.215655 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:55.215640    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:36:55.218178 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:55.218152    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 17 17:36:55.218178 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:55.218157    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 17 17:36:55.218178 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:55.218193    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-c46kfct83tn4k\""
Sep 17 17:36:55.218397 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:55.218203    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 17 17:36:55.218397 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:55.218286    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 17 17:36:55.218397 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:55.218320    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-6cbsr\""
Sep 17 17:36:55.218593 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:55.218578    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 17 17:36:55.218660 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:55.218597    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 17 17:36:55.218660 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:55.218582    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 17 17:36:55.218759 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:55.218665    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 17 17:36:55.218759 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:55.218597    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 17 17:36:55.218759 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:55.218577    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 17 17:36:55.218901 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:55.218885    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 17 17:36:55.219006 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:55.218980    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 17 17:36:55.222351 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:55.222293    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 17 17:36:55.224438 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:55.224420    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 17 17:36:55.237721 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:55.237702    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 17 17:36:55.263030 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:55.263009    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/de34e4c6-ea2c-4c68-b920-d39c07ac860d-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"de34e4c6-ea2c-4c68-b920-d39c07ac860d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:36:55.263132 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:55.263037    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/de34e4c6-ea2c-4c68-b920-d39c07ac860d-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"de34e4c6-ea2c-4c68-b920-d39c07ac860d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:36:55.263132 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:55.263079    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/de34e4c6-ea2c-4c68-b920-d39c07ac860d-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"de34e4c6-ea2c-4c68-b920-d39c07ac860d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:36:55.263132 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:55.263109    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/de34e4c6-ea2c-4c68-b920-d39c07ac860d-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"de34e4c6-ea2c-4c68-b920-d39c07ac860d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:36:55.263234 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:55.263131    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/de34e4c6-ea2c-4c68-b920-d39c07ac860d-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"de34e4c6-ea2c-4c68-b920-d39c07ac860d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:36:55.263234 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:55.263149    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/de34e4c6-ea2c-4c68-b920-d39c07ac860d-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"de34e4c6-ea2c-4c68-b920-d39c07ac860d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:36:55.263234 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:55.263166    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/de34e4c6-ea2c-4c68-b920-d39c07ac860d-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"de34e4c6-ea2c-4c68-b920-d39c07ac860d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:36:55.263234 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:55.263208    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/de34e4c6-ea2c-4c68-b920-d39c07ac860d-web-config\") pod \"prometheus-k8s-0\" (UID: \"de34e4c6-ea2c-4c68-b920-d39c07ac860d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:36:55.263365 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:55.263250    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/de34e4c6-ea2c-4c68-b920-d39c07ac860d-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"de34e4c6-ea2c-4c68-b920-d39c07ac860d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:36:55.263365 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:55.263267    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/de34e4c6-ea2c-4c68-b920-d39c07ac860d-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"de34e4c6-ea2c-4c68-b920-d39c07ac860d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:36:55.263365 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:55.263283    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ptqn7\" (UniqueName: \"kubernetes.io/projected/de34e4c6-ea2c-4c68-b920-d39c07ac860d-kube-api-access-ptqn7\") pod \"prometheus-k8s-0\" (UID: \"de34e4c6-ea2c-4c68-b920-d39c07ac860d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:36:55.263365 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:55.263327    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/de34e4c6-ea2c-4c68-b920-d39c07ac860d-config\") pod \"prometheus-k8s-0\" (UID: \"de34e4c6-ea2c-4c68-b920-d39c07ac860d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:36:55.263365 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:55.263342    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/de34e4c6-ea2c-4c68-b920-d39c07ac860d-config-out\") pod \"prometheus-k8s-0\" (UID: \"de34e4c6-ea2c-4c68-b920-d39c07ac860d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:36:55.263365 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:55.263357    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/de34e4c6-ea2c-4c68-b920-d39c07ac860d-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"de34e4c6-ea2c-4c68-b920-d39c07ac860d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:36:55.263565 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:55.263381    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/de34e4c6-ea2c-4c68-b920-d39c07ac860d-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"de34e4c6-ea2c-4c68-b920-d39c07ac860d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:36:55.263565 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:55.263415    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/de34e4c6-ea2c-4c68-b920-d39c07ac860d-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"de34e4c6-ea2c-4c68-b920-d39c07ac860d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:36:55.263565 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:55.263432    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/de34e4c6-ea2c-4c68-b920-d39c07ac860d-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"de34e4c6-ea2c-4c68-b920-d39c07ac860d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:36:55.263565 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:55.263452    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/de34e4c6-ea2c-4c68-b920-d39c07ac860d-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"de34e4c6-ea2c-4c68-b920-d39c07ac860d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:36:55.263565 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:55.263471    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/de34e4c6-ea2c-4c68-b920-d39c07ac860d-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"de34e4c6-ea2c-4c68-b920-d39c07ac860d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:36:55.263565 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:55.263497    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/de34e4c6-ea2c-4c68-b920-d39c07ac860d-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"de34e4c6-ea2c-4c68-b920-d39c07ac860d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:36:55.363852 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:55.363828    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/de34e4c6-ea2c-4c68-b920-d39c07ac860d-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"de34e4c6-ea2c-4c68-b920-d39c07ac860d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:36:55.363944 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:55.363857    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/de34e4c6-ea2c-4c68-b920-d39c07ac860d-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"de34e4c6-ea2c-4c68-b920-d39c07ac860d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:36:55.363944 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:55.363884    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/de34e4c6-ea2c-4c68-b920-d39c07ac860d-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"de34e4c6-ea2c-4c68-b920-d39c07ac860d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:36:55.363944 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:55.363899    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/de34e4c6-ea2c-4c68-b920-d39c07ac860d-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"de34e4c6-ea2c-4c68-b920-d39c07ac860d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:36:55.363944 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:55.363912    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/de34e4c6-ea2c-4c68-b920-d39c07ac860d-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"de34e4c6-ea2c-4c68-b920-d39c07ac860d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:36:55.364206 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:55.364018    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/de34e4c6-ea2c-4c68-b920-d39c07ac860d-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"de34e4c6-ea2c-4c68-b920-d39c07ac860d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:36:55.364206 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:55.364099    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/de34e4c6-ea2c-4c68-b920-d39c07ac860d-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"de34e4c6-ea2c-4c68-b920-d39c07ac860d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:36:55.364206 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:55.364128    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/de34e4c6-ea2c-4c68-b920-d39c07ac860d-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"de34e4c6-ea2c-4c68-b920-d39c07ac860d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:36:55.364206 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:55.364156    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/de34e4c6-ea2c-4c68-b920-d39c07ac860d-web-config\") pod \"prometheus-k8s-0\" (UID: \"de34e4c6-ea2c-4c68-b920-d39c07ac860d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:36:55.364206 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:55.364192    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/de34e4c6-ea2c-4c68-b920-d39c07ac860d-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"de34e4c6-ea2c-4c68-b920-d39c07ac860d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:36:55.364437 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:55.364219    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/de34e4c6-ea2c-4c68-b920-d39c07ac860d-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"de34e4c6-ea2c-4c68-b920-d39c07ac860d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:36:55.364437 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:55.364247    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ptqn7\" (UniqueName: \"kubernetes.io/projected/de34e4c6-ea2c-4c68-b920-d39c07ac860d-kube-api-access-ptqn7\") pod \"prometheus-k8s-0\" (UID: \"de34e4c6-ea2c-4c68-b920-d39c07ac860d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:36:55.364437 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:55.364276    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/de34e4c6-ea2c-4c68-b920-d39c07ac860d-config\") pod \"prometheus-k8s-0\" (UID: \"de34e4c6-ea2c-4c68-b920-d39c07ac860d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:36:55.364437 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:55.364301    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/de34e4c6-ea2c-4c68-b920-d39c07ac860d-config-out\") pod \"prometheus-k8s-0\" (UID: \"de34e4c6-ea2c-4c68-b920-d39c07ac860d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:36:55.364437 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:55.364328    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/de34e4c6-ea2c-4c68-b920-d39c07ac860d-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"de34e4c6-ea2c-4c68-b920-d39c07ac860d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:36:55.364437 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:55.364367    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/de34e4c6-ea2c-4c68-b920-d39c07ac860d-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"de34e4c6-ea2c-4c68-b920-d39c07ac860d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:36:55.364437 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:55.364421    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/de34e4c6-ea2c-4c68-b920-d39c07ac860d-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"de34e4c6-ea2c-4c68-b920-d39c07ac860d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:36:55.364777 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:55.364454    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/de34e4c6-ea2c-4c68-b920-d39c07ac860d-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"de34e4c6-ea2c-4c68-b920-d39c07ac860d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:36:55.364777 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:55.364477    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/de34e4c6-ea2c-4c68-b920-d39c07ac860d-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"de34e4c6-ea2c-4c68-b920-d39c07ac860d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:36:55.364777 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:55.364500    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/de34e4c6-ea2c-4c68-b920-d39c07ac860d-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"de34e4c6-ea2c-4c68-b920-d39c07ac860d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:36:55.364777 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:55.364694    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/de34e4c6-ea2c-4c68-b920-d39c07ac860d-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"de34e4c6-ea2c-4c68-b920-d39c07ac860d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:36:55.364777 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:55.364703    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/de34e4c6-ea2c-4c68-b920-d39c07ac860d-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"de34e4c6-ea2c-4c68-b920-d39c07ac860d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:36:55.364777 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:55.364711    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/de34e4c6-ea2c-4c68-b920-d39c07ac860d-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"de34e4c6-ea2c-4c68-b920-d39c07ac860d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:36:55.365248 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:55.365223    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/de34e4c6-ea2c-4c68-b920-d39c07ac860d-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"de34e4c6-ea2c-4c68-b920-d39c07ac860d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:36:55.366016 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:55.365722    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/de34e4c6-ea2c-4c68-b920-d39c07ac860d-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"de34e4c6-ea2c-4c68-b920-d39c07ac860d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:36:55.366960 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:55.366936    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/de34e4c6-ea2c-4c68-b920-d39c07ac860d-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"de34e4c6-ea2c-4c68-b920-d39c07ac860d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:36:55.367195 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:55.367164    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/de34e4c6-ea2c-4c68-b920-d39c07ac860d-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"de34e4c6-ea2c-4c68-b920-d39c07ac860d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:36:55.367285 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:55.367250    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/de34e4c6-ea2c-4c68-b920-d39c07ac860d-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"de34e4c6-ea2c-4c68-b920-d39c07ac860d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:36:55.367350 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:55.367277    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/de34e4c6-ea2c-4c68-b920-d39c07ac860d-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"de34e4c6-ea2c-4c68-b920-d39c07ac860d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:36:55.367350 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:55.367302    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/de34e4c6-ea2c-4c68-b920-d39c07ac860d-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"de34e4c6-ea2c-4c68-b920-d39c07ac860d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:36:55.367451 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:55.367345    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/de34e4c6-ea2c-4c68-b920-d39c07ac860d-config\") pod \"prometheus-k8s-0\" (UID: \"de34e4c6-ea2c-4c68-b920-d39c07ac860d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:36:55.367604 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:55.367549    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/de34e4c6-ea2c-4c68-b920-d39c07ac860d-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"de34e4c6-ea2c-4c68-b920-d39c07ac860d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:36:55.367876 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:55.367844    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/de34e4c6-ea2c-4c68-b920-d39c07ac860d-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"de34e4c6-ea2c-4c68-b920-d39c07ac860d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:36:55.368114 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:55.368091    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/de34e4c6-ea2c-4c68-b920-d39c07ac860d-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"de34e4c6-ea2c-4c68-b920-d39c07ac860d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:36:55.368221 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:55.368202    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/de34e4c6-ea2c-4c68-b920-d39c07ac860d-config-out\") pod \"prometheus-k8s-0\" (UID: \"de34e4c6-ea2c-4c68-b920-d39c07ac860d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:36:55.369487 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:55.369463    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/de34e4c6-ea2c-4c68-b920-d39c07ac860d-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"de34e4c6-ea2c-4c68-b920-d39c07ac860d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:36:55.369658 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:55.369644    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/de34e4c6-ea2c-4c68-b920-d39c07ac860d-web-config\") pod \"prometheus-k8s-0\" (UID: \"de34e4c6-ea2c-4c68-b920-d39c07ac860d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:36:55.369786 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:55.369771    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/de34e4c6-ea2c-4c68-b920-d39c07ac860d-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"de34e4c6-ea2c-4c68-b920-d39c07ac860d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:36:55.370529 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:55.370512    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/de34e4c6-ea2c-4c68-b920-d39c07ac860d-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"de34e4c6-ea2c-4c68-b920-d39c07ac860d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:36:55.376314 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:55.376299    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ptqn7\" (UniqueName: \"kubernetes.io/projected/de34e4c6-ea2c-4c68-b920-d39c07ac860d-kube-api-access-ptqn7\") pod \"prometheus-k8s-0\" (UID: \"de34e4c6-ea2c-4c68-b920-d39c07ac860d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:36:55.525322 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:55.525283    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:36:55.815809 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:55.815778    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 17 17:36:55.817455 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:36:55.817430    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde34e4c6_ea2c_4c68_b920_d39c07ac860d.slice/crio-054e609ba2c1c9a99fcdbccb4e890bec4ae5fb16033c226413bf468c18578748 WatchSource:0}: Error finding container 054e609ba2c1c9a99fcdbccb4e890bec4ae5fb16033c226413bf468c18578748: Status 404 returned error can't find the container with id 054e609ba2c1c9a99fcdbccb4e890bec4ae5fb16033c226413bf468c18578748
Sep 17 17:36:55.847669 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:55.847642    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"de34e4c6-ea2c-4c68-b920-d39c07ac860d","Type":"ContainerStarted","Data":"054e609ba2c1c9a99fcdbccb4e890bec4ae5fb16033c226413bf468c18578748"}
Sep 17 17:36:56.853463 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:56.853380    2574 generic.go:356] "Generic (PLEG): container finished" podID="de34e4c6-ea2c-4c68-b920-d39c07ac860d" containerID="69df499065a98255de2a75d2373127abadce4fe2588ee2b40720cdaffa11bd4b" exitCode=0
Sep 17 17:36:56.853784 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:56.853467    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"de34e4c6-ea2c-4c68-b920-d39c07ac860d","Type":"ContainerDied","Data":"69df499065a98255de2a75d2373127abadce4fe2588ee2b40720cdaffa11bd4b"}
Sep 17 17:36:56.953861 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:56.953832    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f2e735a3-7d6c-48d9-84a6-40e168ba42ea" path="/var/lib/kubelet/pods/f2e735a3-7d6c-48d9-84a6-40e168ba42ea/volumes"
Sep 17 17:36:57.860181 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:57.860150    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"de34e4c6-ea2c-4c68-b920-d39c07ac860d","Type":"ContainerStarted","Data":"286d6687e150761961d307b767936c7b461c7ef3275762031518e2d8258f58c0"}
Sep 17 17:36:57.860535 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:57.860215    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"de34e4c6-ea2c-4c68-b920-d39c07ac860d","Type":"ContainerStarted","Data":"92b84989399525594a31fbb0ba3187e067a6170a47e780659d821109ccbef251"}
Sep 17 17:36:57.860535 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:57.860244    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"de34e4c6-ea2c-4c68-b920-d39c07ac860d","Type":"ContainerStarted","Data":"aa7fbfd674e40fb643161615db6a0575b3a98ba557a294c06cafdbc3ae3a618f"}
Sep 17 17:36:57.860535 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:57.860274    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"de34e4c6-ea2c-4c68-b920-d39c07ac860d","Type":"ContainerStarted","Data":"1463ec89ca5374ea3cf49462b9a8c198f9fcbd6fac12cca67ec6835e3a760a24"}
Sep 17 17:36:57.860535 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:57.860298    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"de34e4c6-ea2c-4c68-b920-d39c07ac860d","Type":"ContainerStarted","Data":"0e4e54a43a945993ab0097c083d176343029af26f79b3c519d1ed04bed5653a2"}
Sep 17 17:36:57.860535 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:57.860329    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"de34e4c6-ea2c-4c68-b920-d39c07ac860d","Type":"ContainerStarted","Data":"06f7a7b8c7023093c9a12eecbbd0161a1bd1005b3072ff89fbca6de5aecfc414"}
Sep 17 17:36:57.872142 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:57.872118    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j4ds_d3f553e8-b416-4d7d-bbef-991a0404b287/konnectivity-agent/0.log" limit=1024
Sep 17 17:36:57.893767 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:57.893719    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.875004364 podStartE2EDuration="2.893705099s" totalImagesPullingTime="0s" totalInitContainerRuntime="18.700735ms" isStatefulPod=false podCreationTimestamp="2026-09-17 17:36:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:36:57.891382303 +0000 UTC m=+173.080256038" watchObservedRunningTime="2026-09-17 17:36:57.893705099 +0000 UTC m=+173.082578805"
Sep 17 17:36:57.970947 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:36:57.970926    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j4ds_d3f553e8-b416-4d7d-bbef-991a0404b287/konnectivity-agent/0.log" limit=1024
Sep 17 17:37:00.526433 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:37:00.526401    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:37:05.604675 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:37:05.604633    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-54df6d68d-464vf"]
Sep 17 17:37:05.716387 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:37:05.716355    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j4ds_d3f553e8-b416-4d7d-bbef-991a0404b287/konnectivity-agent/0.log" limit=1024
Sep 17 17:37:05.816176 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:37:05.816146    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j4ds_d3f553e8-b416-4d7d-bbef-991a0404b287/konnectivity-agent/0.log" limit=1024
Sep 17 17:37:05.898018 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:37:05.897947    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j4ds_d3f553e8-b416-4d7d-bbef-991a0404b287/konnectivity-agent/0.log" limit=1024
Sep 17 17:37:11.895674 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:37:11.895641    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j4ds_d3f553e8-b416-4d7d-bbef-991a0404b287/konnectivity-agent/0.log" limit=1024
Sep 17 17:37:13.602383 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:37:13.602356    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j4ds_d3f553e8-b416-4d7d-bbef-991a0404b287/konnectivity-agent/0.log" limit=1024
Sep 17 17:37:13.745905 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:37:13.745874    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j4ds_d3f553e8-b416-4d7d-bbef-991a0404b287/konnectivity-agent/0.log" limit=1024
Sep 17 17:37:30.623052 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:37:30.623006    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-54df6d68d-464vf" podUID="1a4569a8-f709-467c-b575-e7b6defddf64" containerName="console" containerID="cri-o://9ee1b989c83817e8ec3288fa7d9f33c43449c4f94d2e27dfda5a9e4d61951016" gracePeriod=15
Sep 17 17:37:30.863030 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:37:30.863008    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-54df6d68d-464vf"
Sep 17 17:37:30.951429 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:37:30.951358    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1a4569a8-f709-467c-b575-e7b6defddf64-service-ca\") pod \"1a4569a8-f709-467c-b575-e7b6defddf64\" (UID: \"1a4569a8-f709-467c-b575-e7b6defddf64\") "
Sep 17 17:37:30.951429 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:37:30.951392    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1a4569a8-f709-467c-b575-e7b6defddf64-console-oauth-config\") pod \"1a4569a8-f709-467c-b575-e7b6defddf64\" (UID: \"1a4569a8-f709-467c-b575-e7b6defddf64\") "
Sep 17 17:37:30.951429 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:37:30.951410    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1a4569a8-f709-467c-b575-e7b6defddf64-console-config\") pod \"1a4569a8-f709-467c-b575-e7b6defddf64\" (UID: \"1a4569a8-f709-467c-b575-e7b6defddf64\") "
Sep 17 17:37:30.951673 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:37:30.951435    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vmr47\" (UniqueName: \"kubernetes.io/projected/1a4569a8-f709-467c-b575-e7b6defddf64-kube-api-access-vmr47\") pod \"1a4569a8-f709-467c-b575-e7b6defddf64\" (UID: \"1a4569a8-f709-467c-b575-e7b6defddf64\") "
Sep 17 17:37:30.951673 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:37:30.951467    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1a4569a8-f709-467c-b575-e7b6defddf64-trusted-ca-bundle\") pod \"1a4569a8-f709-467c-b575-e7b6defddf64\" (UID: \"1a4569a8-f709-467c-b575-e7b6defddf64\") "
Sep 17 17:37:30.951673 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:37:30.951498    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1a4569a8-f709-467c-b575-e7b6defddf64-oauth-serving-cert\") pod \"1a4569a8-f709-467c-b575-e7b6defddf64\" (UID: \"1a4569a8-f709-467c-b575-e7b6defddf64\") "
Sep 17 17:37:30.951673 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:37:30.951527    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1a4569a8-f709-467c-b575-e7b6defddf64-console-serving-cert\") pod \"1a4569a8-f709-467c-b575-e7b6defddf64\" (UID: \"1a4569a8-f709-467c-b575-e7b6defddf64\") "
Sep 17 17:37:30.952107 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:37:30.952079    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a4569a8-f709-467c-b575-e7b6defddf64-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "1a4569a8-f709-467c-b575-e7b6defddf64" (UID: "1a4569a8-f709-467c-b575-e7b6defddf64"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:37:30.952222 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:37:30.952077    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a4569a8-f709-467c-b575-e7b6defddf64-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "1a4569a8-f709-467c-b575-e7b6defddf64" (UID: "1a4569a8-f709-467c-b575-e7b6defddf64"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:37:30.952222 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:37:30.952182    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a4569a8-f709-467c-b575-e7b6defddf64-service-ca" (OuterVolumeSpecName: "service-ca") pod "1a4569a8-f709-467c-b575-e7b6defddf64" (UID: "1a4569a8-f709-467c-b575-e7b6defddf64"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:37:30.952321 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:37:30.952246    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a4569a8-f709-467c-b575-e7b6defddf64-console-config" (OuterVolumeSpecName: "console-config") pod "1a4569a8-f709-467c-b575-e7b6defddf64" (UID: "1a4569a8-f709-467c-b575-e7b6defddf64"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:37:30.953679 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:37:30.953657    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1a4569a8-f709-467c-b575-e7b6defddf64-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "1a4569a8-f709-467c-b575-e7b6defddf64" (UID: "1a4569a8-f709-467c-b575-e7b6defddf64"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:37:30.953957 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:37:30.953928    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1a4569a8-f709-467c-b575-e7b6defddf64-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "1a4569a8-f709-467c-b575-e7b6defddf64" (UID: "1a4569a8-f709-467c-b575-e7b6defddf64"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:37:30.953957 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:37:30.953938    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a4569a8-f709-467c-b575-e7b6defddf64-kube-api-access-vmr47" (OuterVolumeSpecName: "kube-api-access-vmr47") pod "1a4569a8-f709-467c-b575-e7b6defddf64" (UID: "1a4569a8-f709-467c-b575-e7b6defddf64"). InnerVolumeSpecName "kube-api-access-vmr47". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:37:30.981825 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:37:30.981803    2574 generic.go:356] "Generic (PLEG): container finished" podID="1a4569a8-f709-467c-b575-e7b6defddf64" containerID="9ee1b989c83817e8ec3288fa7d9f33c43449c4f94d2e27dfda5a9e4d61951016" exitCode=0
Sep 17 17:37:30.981915 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:37:30.981838    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-54df6d68d-464vf"
Sep 17 17:37:30.981915 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:37:30.981853    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-54df6d68d-464vf" event={"ID":"1a4569a8-f709-467c-b575-e7b6defddf64","Type":"ContainerDied","Data":"9ee1b989c83817e8ec3288fa7d9f33c43449c4f94d2e27dfda5a9e4d61951016"}
Sep 17 17:37:30.981915 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:37:30.981888    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-54df6d68d-464vf" event={"ID":"1a4569a8-f709-467c-b575-e7b6defddf64","Type":"ContainerDied","Data":"b2f3531e73c75098665a4d01f88cd92894c91c4179b65c1b5421eceeee8190f7"}
Sep 17 17:37:30.981915 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:37:30.981907    2574 scope.go:117] "RemoveContainer" containerID="9ee1b989c83817e8ec3288fa7d9f33c43449c4f94d2e27dfda5a9e4d61951016"
Sep 17 17:37:30.989815 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:37:30.989794    2574 scope.go:117] "RemoveContainer" containerID="9ee1b989c83817e8ec3288fa7d9f33c43449c4f94d2e27dfda5a9e4d61951016"
Sep 17 17:37:30.990086 ip-10-0-134-96 kubenswrapper[2574]: E0917 17:37:30.990065    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ee1b989c83817e8ec3288fa7d9f33c43449c4f94d2e27dfda5a9e4d61951016\": container with ID starting with 9ee1b989c83817e8ec3288fa7d9f33c43449c4f94d2e27dfda5a9e4d61951016 not found: ID does not exist" containerID="9ee1b989c83817e8ec3288fa7d9f33c43449c4f94d2e27dfda5a9e4d61951016"
Sep 17 17:37:30.990164 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:37:30.990091    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ee1b989c83817e8ec3288fa7d9f33c43449c4f94d2e27dfda5a9e4d61951016"} err="failed to get container status \"9ee1b989c83817e8ec3288fa7d9f33c43449c4f94d2e27dfda5a9e4d61951016\": rpc error: code = NotFound desc = could not find container \"9ee1b989c83817e8ec3288fa7d9f33c43449c4f94d2e27dfda5a9e4d61951016\": container with ID starting with 9ee1b989c83817e8ec3288fa7d9f33c43449c4f94d2e27dfda5a9e4d61951016 not found: ID does not exist"
Sep 17 17:37:31.009337 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:37:31.009312    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-54df6d68d-464vf"]
Sep 17 17:37:31.019739 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:37:31.019721    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-54df6d68d-464vf"]
Sep 17 17:37:31.052221 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:37:31.052202    2574 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1a4569a8-f709-467c-b575-e7b6defddf64-service-ca\") on node \"ip-10-0-134-96.ec2.internal\" DevicePath \"\""
Sep 17 17:37:31.052221 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:37:31.052219    2574 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1a4569a8-f709-467c-b575-e7b6defddf64-console-oauth-config\") on node \"ip-10-0-134-96.ec2.internal\" DevicePath \"\""
Sep 17 17:37:31.052323 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:37:31.052228    2574 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1a4569a8-f709-467c-b575-e7b6defddf64-console-config\") on node \"ip-10-0-134-96.ec2.internal\" DevicePath \"\""
Sep 17 17:37:31.052323 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:37:31.052243    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vmr47\" (UniqueName: \"kubernetes.io/projected/1a4569a8-f709-467c-b575-e7b6defddf64-kube-api-access-vmr47\") on node \"ip-10-0-134-96.ec2.internal\" DevicePath \"\""
Sep 17 17:37:31.052323 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:37:31.052252    2574 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1a4569a8-f709-467c-b575-e7b6defddf64-trusted-ca-bundle\") on node \"ip-10-0-134-96.ec2.internal\" DevicePath \"\""
Sep 17 17:37:31.052323 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:37:31.052260    2574 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1a4569a8-f709-467c-b575-e7b6defddf64-oauth-serving-cert\") on node \"ip-10-0-134-96.ec2.internal\" DevicePath \"\""
Sep 17 17:37:31.052323 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:37:31.052268    2574 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1a4569a8-f709-467c-b575-e7b6defddf64-console-serving-cert\") on node \"ip-10-0-134-96.ec2.internal\" DevicePath \"\""
Sep 17 17:37:32.952767 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:37:32.952730    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1a4569a8-f709-467c-b575-e7b6defddf64" path="/var/lib/kubelet/pods/1a4569a8-f709-467c-b575-e7b6defddf64/volumes"
Sep 17 17:37:38.438500 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:37:38.438464    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j4ds_d3f553e8-b416-4d7d-bbef-991a0404b287/konnectivity-agent/0.log" limit=1024
Sep 17 17:37:38.544820 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:37:38.544765    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j4ds_d3f553e8-b416-4d7d-bbef-991a0404b287/konnectivity-agent/0.log" limit=1024
Sep 17 17:37:55.525491 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:37:55.525454    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:37:55.540993 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:37:55.540968    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:37:56.095024 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:37:56.090881    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:38:05.612838 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:38:05.612806    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j4ds_d3f553e8-b416-4d7d-bbef-991a0404b287/konnectivity-agent/0.log" limit=1024
Sep 17 17:38:05.847505 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:38:05.847472    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j4ds_d3f553e8-b416-4d7d-bbef-991a0404b287/konnectivity-agent/0.log" limit=1024
Sep 17 17:38:05.949900 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:38:05.949831    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j4ds_d3f553e8-b416-4d7d-bbef-991a0404b287/konnectivity-agent/0.log" limit=1024
Sep 17 17:38:06.051031 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:38:06.051001    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j4ds_d3f553e8-b416-4d7d-bbef-991a0404b287/konnectivity-agent/0.log" limit=1024
Sep 17 17:38:15.371586 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:38:15.371556    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j4ds_d3f553e8-b416-4d7d-bbef-991a0404b287/konnectivity-agent/0.log" limit=1024
Sep 17 17:38:15.472423 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:38:15.472389    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j4ds_d3f553e8-b416-4d7d-bbef-991a0404b287/konnectivity-agent/0.log" limit=1024
Sep 17 17:38:44.522775 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:38:44.522745    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j4ds_d3f553e8-b416-4d7d-bbef-991a0404b287/konnectivity-agent/0.log" limit=1024
Sep 17 17:38:44.616691 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:38:44.616665    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j4ds_d3f553e8-b416-4d7d-bbef-991a0404b287/konnectivity-agent/0.log" limit=1024
Sep 17 17:38:50.556881 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:38:50.556848    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j4ds_d3f553e8-b416-4d7d-bbef-991a0404b287/konnectivity-agent/0.log" limit=1024
Sep 17 17:38:50.853152 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:38:50.853073    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j4ds_d3f553e8-b416-4d7d-bbef-991a0404b287/konnectivity-agent/0.log" limit=1024
Sep 17 17:38:51.053847 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:38:51.053817    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j4ds_d3f553e8-b416-4d7d-bbef-991a0404b287/konnectivity-agent/0.log" limit=1024
Sep 17 17:38:51.272423 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:38:51.272388    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j4ds_d3f553e8-b416-4d7d-bbef-991a0404b287/konnectivity-agent/0.log" limit=1024
Sep 17 17:38:51.378916 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:38:51.378885    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j4ds_d3f553e8-b416-4d7d-bbef-991a0404b287/konnectivity-agent/0.log" limit=1024
Sep 17 17:38:51.760353 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:38:51.760324    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j4ds_d3f553e8-b416-4d7d-bbef-991a0404b287/konnectivity-agent/0.log" limit=1024
Sep 17 17:38:51.858092 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:38:51.858067    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j4ds_d3f553e8-b416-4d7d-bbef-991a0404b287/konnectivity-agent/0.log" limit=1024
Sep 17 17:38:51.956671 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:38:51.956621    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j4ds_d3f553e8-b416-4d7d-bbef-991a0404b287/konnectivity-agent/0.log" limit=1024
Sep 17 17:38:52.057273 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:38:52.057205    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j4ds_d3f553e8-b416-4d7d-bbef-991a0404b287/konnectivity-agent/0.log" limit=1024
Sep 17 17:38:52.256873 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:38:52.256844    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j4ds_d3f553e8-b416-4d7d-bbef-991a0404b287/konnectivity-agent/0.log" limit=1024
Sep 17 17:38:52.357671 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:38:52.357602    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j4ds_d3f553e8-b416-4d7d-bbef-991a0404b287/konnectivity-agent/0.log" limit=1024
Sep 17 17:39:06.797353 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:39:06.797325    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-45dp8_bd527736-9920-4c79-9069-1eb8be07f869/ovn-acl-logging/0.log"
Sep 17 17:39:06.802550 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:39:06.802530    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-45dp8_bd527736-9920-4c79-9069-1eb8be07f869/ovn-acl-logging/0.log"
Sep 17 17:39:06.809090 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:39:06.809067    2574 kubelet.go:1648] "Image garbage collection succeeded"
Sep 17 17:39:41.923823 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:39:41.923791    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j4ds_d3f553e8-b416-4d7d-bbef-991a0404b287/konnectivity-agent/0.log" limit=1024
Sep 17 17:39:43.299788 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:39:43.299753    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j4ds_d3f553e8-b416-4d7d-bbef-991a0404b287/konnectivity-agent/0.log" limit=1024
Sep 17 17:40:06.576283 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:40:06.576197    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j4ds_d3f553e8-b416-4d7d-bbef-991a0404b287/konnectivity-agent/0.log" limit=1024
Sep 17 17:40:06.808604 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:40:06.808551    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j4ds_d3f553e8-b416-4d7d-bbef-991a0404b287/konnectivity-agent/0.log" limit=1024
Sep 17 17:40:06.905940 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:40:06.905878    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j4ds_d3f553e8-b416-4d7d-bbef-991a0404b287/konnectivity-agent/0.log" limit=1024
Sep 17 17:40:07.008794 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:40:07.008764    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j4ds_d3f553e8-b416-4d7d-bbef-991a0404b287/konnectivity-agent/0.log" limit=1024
Sep 17 17:40:07.485276 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:40:07.485219    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j4ds_d3f553e8-b416-4d7d-bbef-991a0404b287/konnectivity-agent/0.log" limit=1024
Sep 17 17:40:07.583704 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:40:07.583659    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j4ds_d3f553e8-b416-4d7d-bbef-991a0404b287/konnectivity-agent/0.log" limit=1024
Sep 17 17:40:15.236670 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:40:15.236596    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j4ds_d3f553e8-b416-4d7d-bbef-991a0404b287/konnectivity-agent/0.log" limit=1024
Sep 17 17:40:15.332890 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:40:15.332856    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j4ds_d3f553e8-b416-4d7d-bbef-991a0404b287/konnectivity-agent/0.log" limit=1024
Sep 17 17:40:30.113425 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:40:30.113395    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j4ds_d3f553e8-b416-4d7d-bbef-991a0404b287/konnectivity-agent/0.log" limit=1024
Sep 17 17:40:30.192672 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:40:30.192643    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j4ds_d3f553e8-b416-4d7d-bbef-991a0404b287/konnectivity-agent/0.log" limit=1024
Sep 17 17:40:41.470643 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:40:41.470608    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j4ds_d3f553e8-b416-4d7d-bbef-991a0404b287/konnectivity-agent/0.log" limit=1024
Sep 17 17:40:41.697035 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:40:41.697005    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j4ds_d3f553e8-b416-4d7d-bbef-991a0404b287/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:09.912395 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:41:09.912361    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j4ds_d3f553e8-b416-4d7d-bbef-991a0404b287/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:10.024795 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:41:10.024759    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j4ds_d3f553e8-b416-4d7d-bbef-991a0404b287/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:23.645443 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:41:23.645369    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j4ds_d3f553e8-b416-4d7d-bbef-991a0404b287/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:23.731677 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:41:23.731634    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j4ds_d3f553e8-b416-4d7d-bbef-991a0404b287/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:23.802779 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:41:23.802744    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j4ds_d3f553e8-b416-4d7d-bbef-991a0404b287/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:23.903436 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:41:23.903368    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j4ds_d3f553e8-b416-4d7d-bbef-991a0404b287/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:24.020187 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:41:24.020164    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j4ds_d3f553e8-b416-4d7d-bbef-991a0404b287/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:24.770994 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:41:24.770959    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j4ds_d3f553e8-b416-4d7d-bbef-991a0404b287/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:24.841293 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:41:24.841263    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j4ds_d3f553e8-b416-4d7d-bbef-991a0404b287/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:40.107948 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:41:40.107917    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-init-lpsq7"]
Sep 17 17:41:40.108340 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:41:40.108229    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1a4569a8-f709-467c-b575-e7b6defddf64" containerName="console"
Sep 17 17:41:40.108340 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:41:40.108241    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="1a4569a8-f709-467c-b575-e7b6defddf64" containerName="console"
Sep 17 17:41:40.108340 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:41:40.108298    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="1a4569a8-f709-467c-b575-e7b6defddf64" containerName="console"
Sep 17 17:41:40.111125 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:41:40.111109    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-lpsq7"
Sep 17 17:41:40.113469 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:41:40.113443    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Sep 17 17:41:40.113584 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:41:40.113479    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Sep 17 17:41:40.113584 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:41:40.113450    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-kjbd5\""
Sep 17 17:41:40.113663 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:41:40.113443    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"hf-token\""
Sep 17 17:41:40.114283 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:41:40.114270    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Sep 17 17:41:40.119148 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:41:40.119128    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-lpsq7"]
Sep 17 17:41:40.127590 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:41:40.127570    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7a2a1fdf-59cd-44bc-a44e-3be744e6b2a4-model-cache\") pod \"s3-init-lpsq7\" (UID: \"7a2a1fdf-59cd-44bc-a44e-3be744e6b2a4\") " pod="kserve/s3-init-lpsq7"
Sep 17 17:41:40.127705 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:41:40.127660    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w4kdm\" (UniqueName: \"kubernetes.io/projected/7a2a1fdf-59cd-44bc-a44e-3be744e6b2a4-kube-api-access-w4kdm\") pod \"s3-init-lpsq7\" (UID: \"7a2a1fdf-59cd-44bc-a44e-3be744e6b2a4\") " pod="kserve/s3-init-lpsq7"
Sep 17 17:41:40.228165 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:41:40.228139    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w4kdm\" (UniqueName: \"kubernetes.io/projected/7a2a1fdf-59cd-44bc-a44e-3be744e6b2a4-kube-api-access-w4kdm\") pod \"s3-init-lpsq7\" (UID: \"7a2a1fdf-59cd-44bc-a44e-3be744e6b2a4\") " pod="kserve/s3-init-lpsq7"
Sep 17 17:41:40.228281 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:41:40.228180    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7a2a1fdf-59cd-44bc-a44e-3be744e6b2a4-model-cache\") pod \"s3-init-lpsq7\" (UID: \"7a2a1fdf-59cd-44bc-a44e-3be744e6b2a4\") " pod="kserve/s3-init-lpsq7"
Sep 17 17:41:40.228524 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:41:40.228507    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7a2a1fdf-59cd-44bc-a44e-3be744e6b2a4-model-cache\") pod \"s3-init-lpsq7\" (UID: \"7a2a1fdf-59cd-44bc-a44e-3be744e6b2a4\") " pod="kserve/s3-init-lpsq7"
Sep 17 17:41:40.237265 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:41:40.237245    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w4kdm\" (UniqueName: \"kubernetes.io/projected/7a2a1fdf-59cd-44bc-a44e-3be744e6b2a4-kube-api-access-w4kdm\") pod \"s3-init-lpsq7\" (UID: \"7a2a1fdf-59cd-44bc-a44e-3be744e6b2a4\") " pod="kserve/s3-init-lpsq7"
Sep 17 17:41:40.420614 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:41:40.420548    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-lpsq7"
Sep 17 17:41:40.690506 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:41:40.690372    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-lpsq7"]
Sep 17 17:41:40.693098 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:41:40.693069    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7a2a1fdf_59cd_44bc_a44e_3be744e6b2a4.slice/crio-2bcf844018dc220a5a341033a3ba31c43d9105a6102bd0e94b0c80d32fa31ede WatchSource:0}: Error finding container 2bcf844018dc220a5a341033a3ba31c43d9105a6102bd0e94b0c80d32fa31ede: Status 404 returned error can't find the container with id 2bcf844018dc220a5a341033a3ba31c43d9105a6102bd0e94b0c80d32fa31ede
Sep 17 17:41:40.695035 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:41:40.695020    2574 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 17:41:40.902474 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:41:40.902446    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-lpsq7" event={"ID":"7a2a1fdf-59cd-44bc-a44e-3be744e6b2a4","Type":"ContainerStarted","Data":"2bcf844018dc220a5a341033a3ba31c43d9105a6102bd0e94b0c80d32fa31ede"}
Sep 17 17:41:44.921133 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:41:44.921102    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-lpsq7" event={"ID":"7a2a1fdf-59cd-44bc-a44e-3be744e6b2a4","Type":"ContainerStarted","Data":"26412b0b4f56ac5444a651bd3ae576c8db21f8efa93525fec0d9caed97b6b2a2"}
Sep 17 17:41:46.931694 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:41:46.931623    2574 generic.go:356] "Generic (PLEG): container finished" podID="7a2a1fdf-59cd-44bc-a44e-3be744e6b2a4" containerID="26412b0b4f56ac5444a651bd3ae576c8db21f8efa93525fec0d9caed97b6b2a2" exitCode=0
Sep 17 17:41:46.932035 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:41:46.931695    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-lpsq7" event={"ID":"7a2a1fdf-59cd-44bc-a44e-3be744e6b2a4","Type":"ContainerDied","Data":"26412b0b4f56ac5444a651bd3ae576c8db21f8efa93525fec0d9caed97b6b2a2"}
Sep 17 17:41:51.956553 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:41:51.956519    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-lpsq7" event={"ID":"7a2a1fdf-59cd-44bc-a44e-3be744e6b2a4","Type":"ContainerStarted","Data":"7e59b280b01842a0ccf46aa6c126da905ad43fc330cfe410fc7f4afa9da0f3e3"}
Sep 17 17:41:51.977252 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:41:51.977201    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-init-lpsq7" podStartSLOduration=1.657071205 podStartE2EDuration="11.977185507s" totalImagesPullingTime="8.562579437s" totalInitContainerRuntime="1.757534865s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:41:40 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:41:51.975448377 +0000 UTC m=+467.164322087" watchObservedRunningTime="2026-09-17 17:41:51.977185507 +0000 UTC m=+467.166059213"
Sep 17 17:42:02.502755 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:42:02.502721    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j4ds_d3f553e8-b416-4d7d-bbef-991a0404b287/konnectivity-agent/0.log" limit=1024
Sep 17 17:42:56.203328 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:42:56.203300    2574 generic.go:356] "Generic (PLEG): container finished" podID="7a2a1fdf-59cd-44bc-a44e-3be744e6b2a4" containerID="7e59b280b01842a0ccf46aa6c126da905ad43fc330cfe410fc7f4afa9da0f3e3" exitCode=0
Sep 17 17:42:56.203737 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:42:56.203349    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-lpsq7" event={"ID":"7a2a1fdf-59cd-44bc-a44e-3be744e6b2a4","Type":"ContainerDied","Data":"7e59b280b01842a0ccf46aa6c126da905ad43fc330cfe410fc7f4afa9da0f3e3"}
Sep 17 17:42:57.324400 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:42:57.324377    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-lpsq7"
Sep 17 17:42:57.440287 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:42:57.440251    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w4kdm\" (UniqueName: \"kubernetes.io/projected/7a2a1fdf-59cd-44bc-a44e-3be744e6b2a4-kube-api-access-w4kdm\") pod \"7a2a1fdf-59cd-44bc-a44e-3be744e6b2a4\" (UID: \"7a2a1fdf-59cd-44bc-a44e-3be744e6b2a4\") "
Sep 17 17:42:57.440287 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:42:57.440291    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7a2a1fdf-59cd-44bc-a44e-3be744e6b2a4-model-cache\") pod \"7a2a1fdf-59cd-44bc-a44e-3be744e6b2a4\" (UID: \"7a2a1fdf-59cd-44bc-a44e-3be744e6b2a4\") "
Sep 17 17:42:57.442501 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:42:57.442471    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a2a1fdf-59cd-44bc-a44e-3be744e6b2a4-kube-api-access-w4kdm" (OuterVolumeSpecName: "kube-api-access-w4kdm") pod "7a2a1fdf-59cd-44bc-a44e-3be744e6b2a4" (UID: "7a2a1fdf-59cd-44bc-a44e-3be744e6b2a4"). InnerVolumeSpecName "kube-api-access-w4kdm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:42:57.458473 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:42:57.458419    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a2a1fdf-59cd-44bc-a44e-3be744e6b2a4-model-cache" (OuterVolumeSpecName: "model-cache") pod "7a2a1fdf-59cd-44bc-a44e-3be744e6b2a4" (UID: "7a2a1fdf-59cd-44bc-a44e-3be744e6b2a4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:42:57.540939 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:42:57.540908    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w4kdm\" (UniqueName: \"kubernetes.io/projected/7a2a1fdf-59cd-44bc-a44e-3be744e6b2a4-kube-api-access-w4kdm\") on node \"ip-10-0-134-96.ec2.internal\" DevicePath \"\""
Sep 17 17:42:57.540939 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:42:57.540937    2574 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7a2a1fdf-59cd-44bc-a44e-3be744e6b2a4-model-cache\") on node \"ip-10-0-134-96.ec2.internal\" DevicePath \"\""
Sep 17 17:42:58.211362 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:42:58.211328    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-lpsq7"
Sep 17 17:42:58.211362 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:42:58.211345    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-lpsq7" event={"ID":"7a2a1fdf-59cd-44bc-a44e-3be744e6b2a4","Type":"ContainerDied","Data":"2bcf844018dc220a5a341033a3ba31c43d9105a6102bd0e94b0c80d32fa31ede"}
Sep 17 17:42:58.211617 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:42:58.211379    2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2bcf844018dc220a5a341033a3ba31c43d9105a6102bd0e94b0c80d32fa31ede"
Sep 17 17:42:58.237670 ip-10-0-134-96 kubenswrapper[2574]: E0917 17:42:58.237646    2574 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7a2a1fdf_59cd_44bc_a44e_3be744e6b2a4.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7a2a1fdf_59cd_44bc_a44e_3be744e6b2a4.slice/crio-2bcf844018dc220a5a341033a3ba31c43d9105a6102bd0e94b0c80d32fa31ede\": RecentStats: unable to find data in memory cache]"
Sep 17 17:42:58.616581 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:42:58.616540    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j4ds_d3f553e8-b416-4d7d-bbef-991a0404b287/konnectivity-agent/0.log" limit=1024
Sep 17 17:42:58.944854 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:42:58.944780    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j4ds_d3f553e8-b416-4d7d-bbef-991a0404b287/konnectivity-agent/0.log" limit=1024
Sep 17 17:43:10.716340 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:43:10.716313    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j4ds_d3f553e8-b416-4d7d-bbef-991a0404b287/konnectivity-agent/0.log" limit=1024
Sep 17 17:43:10.730560 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:43:10.730532    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-76cbf6b8b7-bnlrp"]
Sep 17 17:43:10.730830 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:43:10.730819    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7a2a1fdf-59cd-44bc-a44e-3be744e6b2a4" containerName="s3-init"
Sep 17 17:43:10.730870 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:43:10.730831    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="7a2a1fdf-59cd-44bc-a44e-3be744e6b2a4" containerName="s3-init"
Sep 17 17:43:10.730870 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:43:10.730843    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7a2a1fdf-59cd-44bc-a44e-3be744e6b2a4" containerName="download-hf-model"
Sep 17 17:43:10.730870 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:43:10.730849    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="7a2a1fdf-59cd-44bc-a44e-3be744e6b2a4" containerName="download-hf-model"
Sep 17 17:43:10.730962 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:43:10.730911    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="7a2a1fdf-59cd-44bc-a44e-3be744e6b2a4" containerName="s3-init"
Sep 17 17:43:10.734161 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:43:10.734146    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-76cbf6b8b7-bnlrp"
Sep 17 17:43:10.736723 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:43:10.736700    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-v2-predictor-serving-cert\""
Sep 17 17:43:10.737064 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:43:10.737034    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 17:43:10.737154 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:43:10.737075    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 17:43:10.737369 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:43:10.737355    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-kube-rbac-proxy-sar-config\""
Sep 17 17:43:10.737934 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:43:10.737916    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-cm7p9\""
Sep 17 17:43:10.738008 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:43:10.737938    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 17:43:10.738071 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:43:10.738008    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 17:43:10.743595 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:43:10.743567    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-76cbf6b8b7-bnlrp"]
Sep 17 17:43:10.840502 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:43:10.840475    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fb568536-6d13-451a-83b9-cb54fa1beb52-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-76cbf6b8b7-bnlrp\" (UID: \"fb568536-6d13-451a-83b9-cb54fa1beb52\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-76cbf6b8b7-bnlrp"
Sep 17 17:43:10.840623 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:43:10.840513    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fb568536-6d13-451a-83b9-cb54fa1beb52-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-v2-predictor-76cbf6b8b7-bnlrp\" (UID: \"fb568536-6d13-451a-83b9-cb54fa1beb52\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-76cbf6b8b7-bnlrp"
Sep 17 17:43:10.840623 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:43:10.840542    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fb568536-6d13-451a-83b9-cb54fa1beb52-kserve-provision-location\") pod \"isvc-canary-rollback-v2-predictor-76cbf6b8b7-bnlrp\" (UID: \"fb568536-6d13-451a-83b9-cb54fa1beb52\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-76cbf6b8b7-bnlrp"
Sep 17 17:43:10.840623 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:43:10.840565    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fb568536-6d13-451a-83b9-cb54fa1beb52-cabundle-cert\") pod \"isvc-canary-rollback-v2-predictor-76cbf6b8b7-bnlrp\" (UID: \"fb568536-6d13-451a-83b9-cb54fa1beb52\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-76cbf6b8b7-bnlrp"
Sep 17 17:43:10.840623 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:43:10.840586    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mpn8z\" (UniqueName: \"kubernetes.io/projected/fb568536-6d13-451a-83b9-cb54fa1beb52-kube-api-access-mpn8z\") pod \"isvc-canary-rollback-v2-predictor-76cbf6b8b7-bnlrp\" (UID: \"fb568536-6d13-451a-83b9-cb54fa1beb52\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-76cbf6b8b7-bnlrp"
Sep 17 17:43:10.843248 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:43:10.843227    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-786544c6c8-l84cg"]
Sep 17 17:43:10.846592 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:43:10.846578    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-786544c6c8-l84cg"
Sep 17 17:43:10.849161 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:43:10.849135    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-predictor-serving-cert\""
Sep 17 17:43:10.852653 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:43:10.852630    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j4ds_d3f553e8-b416-4d7d-bbef-991a0404b287/konnectivity-agent/0.log" limit=1024
Sep 17 17:43:10.857667 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:43:10.857647    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-786544c6c8-l84cg"]
Sep 17 17:43:10.941411 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:43:10.941384    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mpn8z\" (UniqueName: \"kubernetes.io/projected/fb568536-6d13-451a-83b9-cb54fa1beb52-kube-api-access-mpn8z\") pod \"isvc-canary-rollback-v2-predictor-76cbf6b8b7-bnlrp\" (UID: \"fb568536-6d13-451a-83b9-cb54fa1beb52\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-76cbf6b8b7-bnlrp"
Sep 17 17:43:10.941560 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:43:10.941418    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2d0dd10a-cafd-491a-aac4-ebe2baf5c52e-proxy-tls\") pod \"isvc-canary-rollback-predictor-786544c6c8-l84cg\" (UID: \"2d0dd10a-cafd-491a-aac4-ebe2baf5c52e\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-786544c6c8-l84cg"
Sep 17 17:43:10.941560 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:43:10.941447    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2d0dd10a-cafd-491a-aac4-ebe2baf5c52e-cabundle-cert\") pod \"isvc-canary-rollback-predictor-786544c6c8-l84cg\" (UID: \"2d0dd10a-cafd-491a-aac4-ebe2baf5c52e\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-786544c6c8-l84cg"
Sep 17 17:43:10.941560 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:43:10.941479    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fb568536-6d13-451a-83b9-cb54fa1beb52-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-76cbf6b8b7-bnlrp\" (UID: \"fb568536-6d13-451a-83b9-cb54fa1beb52\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-76cbf6b8b7-bnlrp"
Sep 17 17:43:10.941560 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:43:10.941520    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2d0dd10a-cafd-491a-aac4-ebe2baf5c52e-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-786544c6c8-l84cg\" (UID: \"2d0dd10a-cafd-491a-aac4-ebe2baf5c52e\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-786544c6c8-l84cg"
Sep 17 17:43:10.941560 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:43:10.941544    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fb568536-6d13-451a-83b9-cb54fa1beb52-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-v2-predictor-76cbf6b8b7-bnlrp\" (UID: \"fb568536-6d13-451a-83b9-cb54fa1beb52\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-76cbf6b8b7-bnlrp"
Sep 17 17:43:10.941813 ip-10-0-134-96 kubenswrapper[2574]: E0917 17:43:10.941579    2574 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-serving-cert: secret "isvc-canary-rollback-v2-predictor-serving-cert" not found
Sep 17 17:43:10.941813 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:43:10.941598    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-knfkm\" (UniqueName: \"kubernetes.io/projected/2d0dd10a-cafd-491a-aac4-ebe2baf5c52e-kube-api-access-knfkm\") pod \"isvc-canary-rollback-predictor-786544c6c8-l84cg\" (UID: \"2d0dd10a-cafd-491a-aac4-ebe2baf5c52e\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-786544c6c8-l84cg"
Sep 17 17:43:10.941813 ip-10-0-134-96 kubenswrapper[2574]: E0917 17:43:10.941637    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fb568536-6d13-451a-83b9-cb54fa1beb52-proxy-tls podName:fb568536-6d13-451a-83b9-cb54fa1beb52 nodeName:}" failed. No retries permitted until 2026-09-17 17:43:11.441619004 +0000 UTC m=+546.630492703 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/fb568536-6d13-451a-83b9-cb54fa1beb52-proxy-tls") pod "isvc-canary-rollback-v2-predictor-76cbf6b8b7-bnlrp" (UID: "fb568536-6d13-451a-83b9-cb54fa1beb52") : secret "isvc-canary-rollback-v2-predictor-serving-cert" not found
Sep 17 17:43:10.941813 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:43:10.941656    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2d0dd10a-cafd-491a-aac4-ebe2baf5c52e-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-786544c6c8-l84cg\" (UID: \"2d0dd10a-cafd-491a-aac4-ebe2baf5c52e\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-786544c6c8-l84cg"
Sep 17 17:43:10.941813 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:43:10.941684    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fb568536-6d13-451a-83b9-cb54fa1beb52-kserve-provision-location\") pod \"isvc-canary-rollback-v2-predictor-76cbf6b8b7-bnlrp\" (UID: \"fb568536-6d13-451a-83b9-cb54fa1beb52\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-76cbf6b8b7-bnlrp"
Sep 17 17:43:10.941813 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:43:10.941733    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fb568536-6d13-451a-83b9-cb54fa1beb52-cabundle-cert\") pod \"isvc-canary-rollback-v2-predictor-76cbf6b8b7-bnlrp\" (UID: \"fb568536-6d13-451a-83b9-cb54fa1beb52\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-76cbf6b8b7-bnlrp"
Sep 17 17:43:10.942009 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:43:10.941949    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fb568536-6d13-451a-83b9-cb54fa1beb52-kserve-provision-location\") pod \"isvc-canary-rollback-v2-predictor-76cbf6b8b7-bnlrp\" (UID: \"fb568536-6d13-451a-83b9-cb54fa1beb52\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-76cbf6b8b7-bnlrp"
Sep 17 17:43:10.942206 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:43:10.942188    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fb568536-6d13-451a-83b9-cb54fa1beb52-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-v2-predictor-76cbf6b8b7-bnlrp\" (UID: \"fb568536-6d13-451a-83b9-cb54fa1beb52\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-76cbf6b8b7-bnlrp"
Sep 17 17:43:10.942300 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:43:10.942285    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fb568536-6d13-451a-83b9-cb54fa1beb52-cabundle-cert\") pod \"isvc-canary-rollback-v2-predictor-76cbf6b8b7-bnlrp\" (UID: \"fb568536-6d13-451a-83b9-cb54fa1beb52\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-76cbf6b8b7-bnlrp"
Sep 17 17:43:10.952244 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:43:10.952225    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mpn8z\" (UniqueName: \"kubernetes.io/projected/fb568536-6d13-451a-83b9-cb54fa1beb52-kube-api-access-mpn8z\") pod \"isvc-canary-rollback-v2-predictor-76cbf6b8b7-bnlrp\" (UID: \"fb568536-6d13-451a-83b9-cb54fa1beb52\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-76cbf6b8b7-bnlrp"
Sep 17 17:43:10.994751 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:43:10.994727    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j4ds_d3f553e8-b416-4d7d-bbef-991a0404b287/konnectivity-agent/0.log" limit=1024
Sep 17 17:43:11.042755 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:43:11.042733    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2d0dd10a-cafd-491a-aac4-ebe2baf5c52e-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-786544c6c8-l84cg\" (UID: \"2d0dd10a-cafd-491a-aac4-ebe2baf5c52e\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-786544c6c8-l84cg"
Sep 17 17:43:11.042863 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:43:11.042765    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-knfkm\" (UniqueName: \"kubernetes.io/projected/2d0dd10a-cafd-491a-aac4-ebe2baf5c52e-kube-api-access-knfkm\") pod \"isvc-canary-rollback-predictor-786544c6c8-l84cg\" (UID: \"2d0dd10a-cafd-491a-aac4-ebe2baf5c52e\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-786544c6c8-l84cg"
Sep 17 17:43:11.042863 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:43:11.042784    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2d0dd10a-cafd-491a-aac4-ebe2baf5c52e-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-786544c6c8-l84cg\" (UID: \"2d0dd10a-cafd-491a-aac4-ebe2baf5c52e\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-786544c6c8-l84cg"
Sep 17 17:43:11.042863 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:43:11.042811    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2d0dd10a-cafd-491a-aac4-ebe2baf5c52e-proxy-tls\") pod \"isvc-canary-rollback-predictor-786544c6c8-l84cg\" (UID: \"2d0dd10a-cafd-491a-aac4-ebe2baf5c52e\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-786544c6c8-l84cg"
Sep 17 17:43:11.042863 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:43:11.042846    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2d0dd10a-cafd-491a-aac4-ebe2baf5c52e-cabundle-cert\") pod \"isvc-canary-rollback-predictor-786544c6c8-l84cg\" (UID: \"2d0dd10a-cafd-491a-aac4-ebe2baf5c52e\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-786544c6c8-l84cg"
Sep 17 17:43:11.043098 ip-10-0-134-96 kubenswrapper[2574]: E0917 17:43:11.042982    2574 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-rollback-predictor-serving-cert: secret "isvc-canary-rollback-predictor-serving-cert" not found
Sep 17 17:43:11.043098 ip-10-0-134-96 kubenswrapper[2574]: E0917 17:43:11.043096    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2d0dd10a-cafd-491a-aac4-ebe2baf5c52e-proxy-tls podName:2d0dd10a-cafd-491a-aac4-ebe2baf5c52e nodeName:}" failed. No retries permitted until 2026-09-17 17:43:11.543077486 +0000 UTC m=+546.731951186 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/2d0dd10a-cafd-491a-aac4-ebe2baf5c52e-proxy-tls") pod "isvc-canary-rollback-predictor-786544c6c8-l84cg" (UID: "2d0dd10a-cafd-491a-aac4-ebe2baf5c52e") : secret "isvc-canary-rollback-predictor-serving-cert" not found
Sep 17 17:43:11.043221 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:43:11.043171    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2d0dd10a-cafd-491a-aac4-ebe2baf5c52e-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-786544c6c8-l84cg\" (UID: \"2d0dd10a-cafd-491a-aac4-ebe2baf5c52e\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-786544c6c8-l84cg"
Sep 17 17:43:11.043525 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:43:11.043506    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2d0dd10a-cafd-491a-aac4-ebe2baf5c52e-cabundle-cert\") pod \"isvc-canary-rollback-predictor-786544c6c8-l84cg\" (UID: \"2d0dd10a-cafd-491a-aac4-ebe2baf5c52e\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-786544c6c8-l84cg"
Sep 17 17:43:11.043599 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:43:11.043512    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2d0dd10a-cafd-491a-aac4-ebe2baf5c52e-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-786544c6c8-l84cg\" (UID: \"2d0dd10a-cafd-491a-aac4-ebe2baf5c52e\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-786544c6c8-l84cg"
Sep 17 17:43:11.055536 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:43:11.055513    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-knfkm\" (UniqueName: \"kubernetes.io/projected/2d0dd10a-cafd-491a-aac4-ebe2baf5c52e-kube-api-access-knfkm\") pod \"isvc-canary-rollback-predictor-786544c6c8-l84cg\" (UID: \"2d0dd10a-cafd-491a-aac4-ebe2baf5c52e\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-786544c6c8-l84cg"
Sep 17 17:43:11.325149 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:43:11.325066    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j4ds_d3f553e8-b416-4d7d-bbef-991a0404b287/konnectivity-agent/0.log" limit=1024
Sep 17 17:43:11.446255 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:43:11.446227    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fb568536-6d13-451a-83b9-cb54fa1beb52-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-76cbf6b8b7-bnlrp\" (UID: \"fb568536-6d13-451a-83b9-cb54fa1beb52\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-76cbf6b8b7-bnlrp"
Sep 17 17:43:11.448564 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:43:11.448545    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fb568536-6d13-451a-83b9-cb54fa1beb52-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-76cbf6b8b7-bnlrp\" (UID: \"fb568536-6d13-451a-83b9-cb54fa1beb52\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-76cbf6b8b7-bnlrp"
Sep 17 17:43:11.547349 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:43:11.547322    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2d0dd10a-cafd-491a-aac4-ebe2baf5c52e-proxy-tls\") pod \"isvc-canary-rollback-predictor-786544c6c8-l84cg\" (UID: \"2d0dd10a-cafd-491a-aac4-ebe2baf5c52e\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-786544c6c8-l84cg"
Sep 17 17:43:11.549542 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:43:11.549521    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2d0dd10a-cafd-491a-aac4-ebe2baf5c52e-proxy-tls\") pod \"isvc-canary-rollback-predictor-786544c6c8-l84cg\" (UID: \"2d0dd10a-cafd-491a-aac4-ebe2baf5c52e\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-786544c6c8-l84cg"
Sep 17 17:43:11.645163 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:43:11.645104    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-76cbf6b8b7-bnlrp"
Sep 17 17:43:11.739683 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:43:11.739655    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j4ds_d3f553e8-b416-4d7d-bbef-991a0404b287/konnectivity-agent/0.log" limit=1024
Sep 17 17:43:11.756881 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:43:11.756856    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-786544c6c8-l84cg"
Sep 17 17:43:11.916570 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:43:11.916544    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-76cbf6b8b7-bnlrp"]
Sep 17 17:43:11.917987 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:43:11.917963    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb568536_6d13_451a_83b9_cb54fa1beb52.slice/crio-ee62349a5eec3ab6fe322b8f312577eb331e3ed3d1501e4e7c11c29a4b2cdef7 WatchSource:0}: Error finding container ee62349a5eec3ab6fe322b8f312577eb331e3ed3d1501e4e7c11c29a4b2cdef7: Status 404 returned error can't find the container with id ee62349a5eec3ab6fe322b8f312577eb331e3ed3d1501e4e7c11c29a4b2cdef7
Sep 17 17:43:12.028195 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:43:12.028123    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-786544c6c8-l84cg"]
Sep 17 17:43:12.030383 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:43:12.030361    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d0dd10a_cafd_491a_aac4_ebe2baf5c52e.slice/crio-b32f01d7761944d4d23fdb84b27915c7dd96438d27faa752432f5523d8e7a174 WatchSource:0}: Error finding container b32f01d7761944d4d23fdb84b27915c7dd96438d27faa752432f5523d8e7a174: Status 404 returned error can't find the container with id b32f01d7761944d4d23fdb84b27915c7dd96438d27faa752432f5523d8e7a174
Sep 17 17:43:12.267124 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:43:12.267086    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-786544c6c8-l84cg" event={"ID":"2d0dd10a-cafd-491a-aac4-ebe2baf5c52e","Type":"ContainerStarted","Data":"3618723be650f18cf5f6b40cc0235738839f7bd6581582d6898fe3c9f5d085d1"}
Sep 17 17:43:12.267318 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:43:12.267159    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-786544c6c8-l84cg" event={"ID":"2d0dd10a-cafd-491a-aac4-ebe2baf5c52e","Type":"ContainerStarted","Data":"b32f01d7761944d4d23fdb84b27915c7dd96438d27faa752432f5523d8e7a174"}
Sep 17 17:43:12.268799 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:43:12.268773    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-76cbf6b8b7-bnlrp" event={"ID":"fb568536-6d13-451a-83b9-cb54fa1beb52","Type":"ContainerStarted","Data":"34ac5ac18e6da82b72d2ed2efaf58e649d8648b48d7a28408a8041013e71ea9b"}
Sep 17 17:43:12.268928 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:43:12.268828    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-76cbf6b8b7-bnlrp" event={"ID":"fb568536-6d13-451a-83b9-cb54fa1beb52","Type":"ContainerStarted","Data":"ee62349a5eec3ab6fe322b8f312577eb331e3ed3d1501e4e7c11c29a4b2cdef7"}
Sep 17 17:43:16.286341 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:43:16.286305    2574 generic.go:356] "Generic (PLEG): container finished" podID="2d0dd10a-cafd-491a-aac4-ebe2baf5c52e" containerID="3618723be650f18cf5f6b40cc0235738839f7bd6581582d6898fe3c9f5d085d1" exitCode=0
Sep 17 17:43:16.286728 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:43:16.286337    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-786544c6c8-l84cg" event={"ID":"2d0dd10a-cafd-491a-aac4-ebe2baf5c52e","Type":"ContainerDied","Data":"3618723be650f18cf5f6b40cc0235738839f7bd6581582d6898fe3c9f5d085d1"}
Sep 17 17:43:16.288147 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:43:16.288123    2574 generic.go:356] "Generic (PLEG): container finished" podID="fb568536-6d13-451a-83b9-cb54fa1beb52" containerID="34ac5ac18e6da82b72d2ed2efaf58e649d8648b48d7a28408a8041013e71ea9b" exitCode=0
Sep 17 17:43:16.288214 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:43:16.288190    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-76cbf6b8b7-bnlrp" event={"ID":"fb568536-6d13-451a-83b9-cb54fa1beb52","Type":"ContainerDied","Data":"34ac5ac18e6da82b72d2ed2efaf58e649d8648b48d7a28408a8041013e71ea9b"}
Sep 17 17:43:19.267793 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:43:19.267759    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j4ds_d3f553e8-b416-4d7d-bbef-991a0404b287/konnectivity-agent/0.log" limit=1024
Sep 17 17:43:33.390725 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:43:33.390694    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-76cbf6b8b7-bnlrp" event={"ID":"fb568536-6d13-451a-83b9-cb54fa1beb52","Type":"ContainerStarted","Data":"8162b99cbe8f583ea08bcaa3b9f4303731f88da4394c606f2a2af22a0b61532b"}
Sep 17 17:43:33.392927 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:43:33.392900    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-786544c6c8-l84cg" event={"ID":"2d0dd10a-cafd-491a-aac4-ebe2baf5c52e","Type":"ContainerStarted","Data":"21e5b44dce66d42d5a751116d64386fb51fdcd38307200bbd041078ea459006c"}
Sep 17 17:43:36.408112 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:43:36.408078    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-76cbf6b8b7-bnlrp" event={"ID":"fb568536-6d13-451a-83b9-cb54fa1beb52","Type":"ContainerStarted","Data":"a4e7b6e86351d00fdec1c812a5ed86ba575641a4b1da8375556f142fd6032841"}
Sep 17 17:43:36.408569 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:43:36.408320    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-76cbf6b8b7-bnlrp"
Sep 17 17:43:36.408569 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:43:36.408371    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-76cbf6b8b7-bnlrp"
Sep 17 17:43:36.409431 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:43:36.409401    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-76cbf6b8b7-bnlrp" podUID="fb568536-6d13-451a-83b9-cb54fa1beb52" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Sep 17 17:43:36.410227 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:43:36.410208    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-786544c6c8-l84cg" event={"ID":"2d0dd10a-cafd-491a-aac4-ebe2baf5c52e","Type":"ContainerStarted","Data":"41066f4f3f58233ed92a04276eaba3684581888362d7c24b602fe992a7987a94"}
Sep 17 17:43:36.410382 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:43:36.410364    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-786544c6c8-l84cg"
Sep 17 17:43:36.430695 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:43:36.430653    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-76cbf6b8b7-bnlrp" podStartSLOduration=3.855423071 podStartE2EDuration="26.430642026s" totalImagesPullingTime="18.961863254s" totalInitContainerRuntime="3.613355701s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:43:10 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:43:36.429370847 +0000 UTC m=+571.618244554" watchObservedRunningTime="2026-09-17 17:43:36.430642026 +0000 UTC m=+571.619515791"
Sep 17 17:43:37.415367 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:43:37.415332    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-76cbf6b8b7-bnlrp" podUID="fb568536-6d13-451a-83b9-cb54fa1beb52" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Sep 17 17:43:37.415743 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:43:37.415451    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-786544c6c8-l84cg"
Sep 17 17:43:37.416227 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:43:37.416202    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-786544c6c8-l84cg" podUID="2d0dd10a-cafd-491a-aac4-ebe2baf5c52e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Sep 17 17:43:38.419000 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:43:38.418960    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-786544c6c8-l84cg" podUID="2d0dd10a-cafd-491a-aac4-ebe2baf5c52e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Sep 17 17:43:42.419436 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:43:42.419404    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-76cbf6b8b7-bnlrp"
Sep 17 17:43:42.419904 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:43:42.419878    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-76cbf6b8b7-bnlrp" podUID="fb568536-6d13-451a-83b9-cb54fa1beb52" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Sep 17 17:43:42.443312 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:43:42.443267    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-786544c6c8-l84cg" podStartSLOduration=9.593123509 podStartE2EDuration="32.443253509s" totalImagesPullingTime="18.991215409s" totalInitContainerRuntime="3.858914591s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:43:10 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:43:36.451425986 +0000 UTC m=+571.640299691" watchObservedRunningTime="2026-09-17 17:43:42.443253509 +0000 UTC m=+577.632127214"
Sep 17 17:43:43.423671 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:43:43.423641    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-786544c6c8-l84cg"
Sep 17 17:43:43.424174 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:43:43.424148    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-786544c6c8-l84cg" podUID="2d0dd10a-cafd-491a-aac4-ebe2baf5c52e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Sep 17 17:43:52.419990 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:43:52.419953    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-76cbf6b8b7-bnlrp" podUID="fb568536-6d13-451a-83b9-cb54fa1beb52" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Sep 17 17:43:53.424967 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:43:53.424927    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-786544c6c8-l84cg" podUID="2d0dd10a-cafd-491a-aac4-ebe2baf5c52e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Sep 17 17:44:02.420808 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:44:02.420766    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-76cbf6b8b7-bnlrp" podUID="fb568536-6d13-451a-83b9-cb54fa1beb52" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Sep 17 17:44:03.425124 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:44:03.425091    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-786544c6c8-l84cg" podUID="2d0dd10a-cafd-491a-aac4-ebe2baf5c52e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Sep 17 17:44:04.830755 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:44:04.830720    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j4ds_d3f553e8-b416-4d7d-bbef-991a0404b287/konnectivity-agent/0.log" limit=1024
Sep 17 17:44:06.830895 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:44:06.830859    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-45dp8_bd527736-9920-4c79-9069-1eb8be07f869/ovn-acl-logging/0.log"
Sep 17 17:44:06.833844 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:44:06.833825    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-45dp8_bd527736-9920-4c79-9069-1eb8be07f869/ovn-acl-logging/0.log"
Sep 17 17:44:12.420454 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:44:12.420413    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-76cbf6b8b7-bnlrp" podUID="fb568536-6d13-451a-83b9-cb54fa1beb52" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Sep 17 17:44:13.424694 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:44:13.424656    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-786544c6c8-l84cg" podUID="2d0dd10a-cafd-491a-aac4-ebe2baf5c52e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Sep 17 17:44:22.420853 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:44:22.420765    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-76cbf6b8b7-bnlrp" podUID="fb568536-6d13-451a-83b9-cb54fa1beb52" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Sep 17 17:44:23.424803 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:44:23.424760    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-786544c6c8-l84cg" podUID="2d0dd10a-cafd-491a-aac4-ebe2baf5c52e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Sep 17 17:44:32.419957 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:44:32.419918    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-76cbf6b8b7-bnlrp" podUID="fb568536-6d13-451a-83b9-cb54fa1beb52" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Sep 17 17:44:33.424572 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:44:33.424537    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-786544c6c8-l84cg" podUID="2d0dd10a-cafd-491a-aac4-ebe2baf5c52e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Sep 17 17:44:42.420283 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:44:42.420240    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-76cbf6b8b7-bnlrp" podUID="fb568536-6d13-451a-83b9-cb54fa1beb52" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Sep 17 17:44:43.425231 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:44:43.425203    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-786544c6c8-l84cg"
Sep 17 17:44:45.949203 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:44:45.949173    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-76cbf6b8b7-bnlrp"
Sep 17 17:44:50.651902 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:44:50.651867    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-76cbf6b8b7-bnlrp"]
Sep 17 17:44:50.652294 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:44:50.652090    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-76cbf6b8b7-bnlrp" podUID="fb568536-6d13-451a-83b9-cb54fa1beb52" containerName="kserve-container" containerID="cri-o://8162b99cbe8f583ea08bcaa3b9f4303731f88da4394c606f2a2af22a0b61532b" gracePeriod=30
Sep 17 17:44:50.652294 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:44:50.652117    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-76cbf6b8b7-bnlrp" podUID="fb568536-6d13-451a-83b9-cb54fa1beb52" containerName="kube-rbac-proxy" containerID="cri-o://a4e7b6e86351d00fdec1c812a5ed86ba575641a4b1da8375556f142fd6032841" gracePeriod=30
Sep 17 17:44:50.701945 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:44:50.701919    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j4ds_d3f553e8-b416-4d7d-bbef-991a0404b287/konnectivity-agent/0.log" limit=1024
Sep 17 17:44:51.701540 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:44:51.701509    2574 generic.go:356] "Generic (PLEG): container finished" podID="fb568536-6d13-451a-83b9-cb54fa1beb52" containerID="a4e7b6e86351d00fdec1c812a5ed86ba575641a4b1da8375556f142fd6032841" exitCode=2
Sep 17 17:44:51.701900 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:44:51.701582    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-76cbf6b8b7-bnlrp" event={"ID":"fb568536-6d13-451a-83b9-cb54fa1beb52","Type":"ContainerDied","Data":"a4e7b6e86351d00fdec1c812a5ed86ba575641a4b1da8375556f142fd6032841"}
Sep 17 17:44:52.416200 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:44:52.416153    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-76cbf6b8b7-bnlrp" podUID="fb568536-6d13-451a-83b9-cb54fa1beb52" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.24:8643/healthz\": dial tcp 10.134.0.24:8643: connect: connection refused"
Sep 17 17:44:55.292106 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:44:55.292082    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-76cbf6b8b7-bnlrp"
Sep 17 17:44:55.407166 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:44:55.407088    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fb568536-6d13-451a-83b9-cb54fa1beb52-proxy-tls\") pod \"fb568536-6d13-451a-83b9-cb54fa1beb52\" (UID: \"fb568536-6d13-451a-83b9-cb54fa1beb52\") "
Sep 17 17:44:55.407166 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:44:55.407150    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fb568536-6d13-451a-83b9-cb54fa1beb52-cabundle-cert\") pod \"fb568536-6d13-451a-83b9-cb54fa1beb52\" (UID: \"fb568536-6d13-451a-83b9-cb54fa1beb52\") "
Sep 17 17:44:55.407375 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:44:55.407182    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mpn8z\" (UniqueName: \"kubernetes.io/projected/fb568536-6d13-451a-83b9-cb54fa1beb52-kube-api-access-mpn8z\") pod \"fb568536-6d13-451a-83b9-cb54fa1beb52\" (UID: \"fb568536-6d13-451a-83b9-cb54fa1beb52\") "
Sep 17 17:44:55.407375 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:44:55.407247    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fb568536-6d13-451a-83b9-cb54fa1beb52-kserve-provision-location\") pod \"fb568536-6d13-451a-83b9-cb54fa1beb52\" (UID: \"fb568536-6d13-451a-83b9-cb54fa1beb52\") "
Sep 17 17:44:55.407375 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:44:55.407277    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fb568536-6d13-451a-83b9-cb54fa1beb52-kube-rbac-proxy-sar-config\") pod \"fb568536-6d13-451a-83b9-cb54fa1beb52\" (UID: \"fb568536-6d13-451a-83b9-cb54fa1beb52\") "
Sep 17 17:44:55.407545 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:44:55.407522    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb568536-6d13-451a-83b9-cb54fa1beb52-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "fb568536-6d13-451a-83b9-cb54fa1beb52" (UID: "fb568536-6d13-451a-83b9-cb54fa1beb52"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:44:55.407607 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:44:55.407578    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb568536-6d13-451a-83b9-cb54fa1beb52-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "fb568536-6d13-451a-83b9-cb54fa1beb52" (UID: "fb568536-6d13-451a-83b9-cb54fa1beb52"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:44:55.407754 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:44:55.407727    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb568536-6d13-451a-83b9-cb54fa1beb52-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "fb568536-6d13-451a-83b9-cb54fa1beb52" (UID: "fb568536-6d13-451a-83b9-cb54fa1beb52"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:44:55.409393 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:44:55.409371    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb568536-6d13-451a-83b9-cb54fa1beb52-kube-api-access-mpn8z" (OuterVolumeSpecName: "kube-api-access-mpn8z") pod "fb568536-6d13-451a-83b9-cb54fa1beb52" (UID: "fb568536-6d13-451a-83b9-cb54fa1beb52"). InnerVolumeSpecName "kube-api-access-mpn8z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:44:55.409507 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:44:55.409489    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb568536-6d13-451a-83b9-cb54fa1beb52-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "fb568536-6d13-451a-83b9-cb54fa1beb52" (UID: "fb568536-6d13-451a-83b9-cb54fa1beb52"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:44:55.508061 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:44:55.508019    2574 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fb568536-6d13-451a-83b9-cb54fa1beb52-proxy-tls\") on node \"ip-10-0-134-96.ec2.internal\" DevicePath \"\""
Sep 17 17:44:55.508061 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:44:55.508057    2574 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fb568536-6d13-451a-83b9-cb54fa1beb52-cabundle-cert\") on node \"ip-10-0-134-96.ec2.internal\" DevicePath \"\""
Sep 17 17:44:55.508061 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:44:55.508068    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mpn8z\" (UniqueName: \"kubernetes.io/projected/fb568536-6d13-451a-83b9-cb54fa1beb52-kube-api-access-mpn8z\") on node \"ip-10-0-134-96.ec2.internal\" DevicePath \"\""
Sep 17 17:44:55.508245 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:44:55.508076    2574 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fb568536-6d13-451a-83b9-cb54fa1beb52-kserve-provision-location\") on node \"ip-10-0-134-96.ec2.internal\" DevicePath \"\""
Sep 17 17:44:55.508245 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:44:55.508085    2574 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fb568536-6d13-451a-83b9-cb54fa1beb52-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-96.ec2.internal\" DevicePath \"\""
Sep 17 17:44:55.720335 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:44:55.720255    2574 generic.go:356] "Generic (PLEG): container finished" podID="fb568536-6d13-451a-83b9-cb54fa1beb52" containerID="8162b99cbe8f583ea08bcaa3b9f4303731f88da4394c606f2a2af22a0b61532b" exitCode=0
Sep 17 17:44:55.720335 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:44:55.720303    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-76cbf6b8b7-bnlrp" event={"ID":"fb568536-6d13-451a-83b9-cb54fa1beb52","Type":"ContainerDied","Data":"8162b99cbe8f583ea08bcaa3b9f4303731f88da4394c606f2a2af22a0b61532b"}
Sep 17 17:44:55.720335 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:44:55.720317    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-76cbf6b8b7-bnlrp"
Sep 17 17:44:55.720532 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:44:55.720352    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-76cbf6b8b7-bnlrp" event={"ID":"fb568536-6d13-451a-83b9-cb54fa1beb52","Type":"ContainerDied","Data":"ee62349a5eec3ab6fe322b8f312577eb331e3ed3d1501e4e7c11c29a4b2cdef7"}
Sep 17 17:44:55.720532 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:44:55.720378    2574 scope.go:117] "RemoveContainer" containerID="a4e7b6e86351d00fdec1c812a5ed86ba575641a4b1da8375556f142fd6032841"
Sep 17 17:44:55.729597 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:44:55.729572    2574 scope.go:117] "RemoveContainer" containerID="8162b99cbe8f583ea08bcaa3b9f4303731f88da4394c606f2a2af22a0b61532b"
Sep 17 17:44:55.737215 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:44:55.737197    2574 scope.go:117] "RemoveContainer" containerID="34ac5ac18e6da82b72d2ed2efaf58e649d8648b48d7a28408a8041013e71ea9b"
Sep 17 17:44:55.743918 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:44:55.743899    2574 scope.go:117] "RemoveContainer" containerID="a4e7b6e86351d00fdec1c812a5ed86ba575641a4b1da8375556f142fd6032841"
Sep 17 17:44:55.744193 ip-10-0-134-96 kubenswrapper[2574]: E0917 17:44:55.744176    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a4e7b6e86351d00fdec1c812a5ed86ba575641a4b1da8375556f142fd6032841\": container with ID starting with a4e7b6e86351d00fdec1c812a5ed86ba575641a4b1da8375556f142fd6032841 not found: ID does not exist" containerID="a4e7b6e86351d00fdec1c812a5ed86ba575641a4b1da8375556f142fd6032841"
Sep 17 17:44:55.744260 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:44:55.744199    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a4e7b6e86351d00fdec1c812a5ed86ba575641a4b1da8375556f142fd6032841"} err="failed to get container status \"a4e7b6e86351d00fdec1c812a5ed86ba575641a4b1da8375556f142fd6032841\": rpc error: code = NotFound desc = could not find container \"a4e7b6e86351d00fdec1c812a5ed86ba575641a4b1da8375556f142fd6032841\": container with ID starting with a4e7b6e86351d00fdec1c812a5ed86ba575641a4b1da8375556f142fd6032841 not found: ID does not exist"
Sep 17 17:44:55.744260 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:44:55.744225    2574 scope.go:117] "RemoveContainer" containerID="8162b99cbe8f583ea08bcaa3b9f4303731f88da4394c606f2a2af22a0b61532b"
Sep 17 17:44:55.744473 ip-10-0-134-96 kubenswrapper[2574]: E0917 17:44:55.744455    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8162b99cbe8f583ea08bcaa3b9f4303731f88da4394c606f2a2af22a0b61532b\": container with ID starting with 8162b99cbe8f583ea08bcaa3b9f4303731f88da4394c606f2a2af22a0b61532b not found: ID does not exist" containerID="8162b99cbe8f583ea08bcaa3b9f4303731f88da4394c606f2a2af22a0b61532b"
Sep 17 17:44:55.744534 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:44:55.744482    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8162b99cbe8f583ea08bcaa3b9f4303731f88da4394c606f2a2af22a0b61532b"} err="failed to get container status \"8162b99cbe8f583ea08bcaa3b9f4303731f88da4394c606f2a2af22a0b61532b\": rpc error: code = NotFound desc = could not find container \"8162b99cbe8f583ea08bcaa3b9f4303731f88da4394c606f2a2af22a0b61532b\": container with ID starting with 8162b99cbe8f583ea08bcaa3b9f4303731f88da4394c606f2a2af22a0b61532b not found: ID does not exist"
Sep 17 17:44:55.744534 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:44:55.744504    2574 scope.go:117] "RemoveContainer" containerID="34ac5ac18e6da82b72d2ed2efaf58e649d8648b48d7a28408a8041013e71ea9b"
Sep 17 17:44:55.744762 ip-10-0-134-96 kubenswrapper[2574]: E0917 17:44:55.744743    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"34ac5ac18e6da82b72d2ed2efaf58e649d8648b48d7a28408a8041013e71ea9b\": container with ID starting with 34ac5ac18e6da82b72d2ed2efaf58e649d8648b48d7a28408a8041013e71ea9b not found: ID does not exist" containerID="34ac5ac18e6da82b72d2ed2efaf58e649d8648b48d7a28408a8041013e71ea9b"
Sep 17 17:44:55.744841 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:44:55.744765    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"34ac5ac18e6da82b72d2ed2efaf58e649d8648b48d7a28408a8041013e71ea9b"} err="failed to get container status \"34ac5ac18e6da82b72d2ed2efaf58e649d8648b48d7a28408a8041013e71ea9b\": rpc error: code = NotFound desc = could not find container \"34ac5ac18e6da82b72d2ed2efaf58e649d8648b48d7a28408a8041013e71ea9b\": container with ID starting with 34ac5ac18e6da82b72d2ed2efaf58e649d8648b48d7a28408a8041013e71ea9b not found: ID does not exist"
Sep 17 17:44:55.752582 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:44:55.752559    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-76cbf6b8b7-bnlrp"]
Sep 17 17:44:55.763483 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:44:55.763463    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-76cbf6b8b7-bnlrp"]
Sep 17 17:44:55.827253 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:44:55.827210    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j4ds_d3f553e8-b416-4d7d-bbef-991a0404b287/konnectivity-agent/0.log" limit=1024
Sep 17 17:44:55.884956 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:44:55.884927    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-636c4-predictor-5879d668f5-6hwtz"]
Sep 17 17:44:55.885264 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:44:55.885251    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fb568536-6d13-451a-83b9-cb54fa1beb52" containerName="kserve-container"
Sep 17 17:44:55.885310 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:44:55.885266    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="fb568536-6d13-451a-83b9-cb54fa1beb52" containerName="kserve-container"
Sep 17 17:44:55.885310 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:44:55.885274    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fb568536-6d13-451a-83b9-cb54fa1beb52" containerName="kube-rbac-proxy"
Sep 17 17:44:55.885310 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:44:55.885279    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="fb568536-6d13-451a-83b9-cb54fa1beb52" containerName="kube-rbac-proxy"
Sep 17 17:44:55.885310 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:44:55.885290    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fb568536-6d13-451a-83b9-cb54fa1beb52" containerName="storage-initializer"
Sep 17 17:44:55.885310 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:44:55.885295    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="fb568536-6d13-451a-83b9-cb54fa1beb52" containerName="storage-initializer"
Sep 17 17:44:55.885455 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:44:55.885341    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="fb568536-6d13-451a-83b9-cb54fa1beb52" containerName="kube-rbac-proxy"
Sep 17 17:44:55.885455 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:44:55.885352    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="fb568536-6d13-451a-83b9-cb54fa1beb52" containerName="kserve-container"
Sep 17 17:44:55.889795 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:44:55.889778    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-636c4-predictor-5879d668f5-6hwtz"
Sep 17 17:44:55.892252 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:44:55.892234    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-636c4-predictor-serving-cert\""
Sep 17 17:44:55.892323 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:44:55.892252    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-636c4-kube-rbac-proxy-sar-config\""
Sep 17 17:44:55.899749 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:44:55.899729    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-636c4-predictor-5879d668f5-6hwtz"]
Sep 17 17:44:56.011938 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:44:56.011909    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d4b9e97c-cfed-4de6-b410-4d8b1f3c98b5-cabundle-cert\") pod \"isvc-sklearn-graph-raw-636c4-predictor-5879d668f5-6hwtz\" (UID: \"d4b9e97c-cfed-4de6-b410-4d8b1f3c98b5\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-636c4-predictor-5879d668f5-6hwtz"
Sep 17 17:44:56.012103 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:44:56.011946    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d4b9e97c-cfed-4de6-b410-4d8b1f3c98b5-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-636c4-predictor-5879d668f5-6hwtz\" (UID: \"d4b9e97c-cfed-4de6-b410-4d8b1f3c98b5\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-636c4-predictor-5879d668f5-6hwtz"
Sep 17 17:44:56.012103 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:44:56.011970    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d4b9e97c-cfed-4de6-b410-4d8b1f3c98b5-proxy-tls\") pod \"isvc-sklearn-graph-raw-636c4-predictor-5879d668f5-6hwtz\" (UID: \"d4b9e97c-cfed-4de6-b410-4d8b1f3c98b5\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-636c4-predictor-5879d668f5-6hwtz"
Sep 17 17:44:56.012103 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:44:56.012036    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n85sp\" (UniqueName: \"kubernetes.io/projected/d4b9e97c-cfed-4de6-b410-4d8b1f3c98b5-kube-api-access-n85sp\") pod \"isvc-sklearn-graph-raw-636c4-predictor-5879d668f5-6hwtz\" (UID: \"d4b9e97c-cfed-4de6-b410-4d8b1f3c98b5\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-636c4-predictor-5879d668f5-6hwtz"
Sep 17 17:44:56.012208 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:44:56.012135    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d4b9e97c-cfed-4de6-b410-4d8b1f3c98b5-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-636c4-predictor-5879d668f5-6hwtz\" (UID: \"d4b9e97c-cfed-4de6-b410-4d8b1f3c98b5\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-636c4-predictor-5879d668f5-6hwtz"
Sep 17 17:44:56.018620 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:44:56.018594    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j4ds_d3f553e8-b416-4d7d-bbef-991a0404b287/konnectivity-agent/0.log" limit=1024
Sep 17 17:44:56.113031 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:44:56.113002    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d4b9e97c-cfed-4de6-b410-4d8b1f3c98b5-cabundle-cert\") pod \"isvc-sklearn-graph-raw-636c4-predictor-5879d668f5-6hwtz\" (UID: \"d4b9e97c-cfed-4de6-b410-4d8b1f3c98b5\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-636c4-predictor-5879d668f5-6hwtz"
Sep 17 17:44:56.113271 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:44:56.113038    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d4b9e97c-cfed-4de6-b410-4d8b1f3c98b5-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-636c4-predictor-5879d668f5-6hwtz\" (UID: \"d4b9e97c-cfed-4de6-b410-4d8b1f3c98b5\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-636c4-predictor-5879d668f5-6hwtz"
Sep 17 17:44:56.113271 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:44:56.113085    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d4b9e97c-cfed-4de6-b410-4d8b1f3c98b5-proxy-tls\") pod \"isvc-sklearn-graph-raw-636c4-predictor-5879d668f5-6hwtz\" (UID: \"d4b9e97c-cfed-4de6-b410-4d8b1f3c98b5\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-636c4-predictor-5879d668f5-6hwtz"
Sep 17 17:44:56.113271 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:44:56.113101    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n85sp\" (UniqueName: \"kubernetes.io/projected/d4b9e97c-cfed-4de6-b410-4d8b1f3c98b5-kube-api-access-n85sp\") pod \"isvc-sklearn-graph-raw-636c4-predictor-5879d668f5-6hwtz\" (UID: \"d4b9e97c-cfed-4de6-b410-4d8b1f3c98b5\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-636c4-predictor-5879d668f5-6hwtz"
Sep 17 17:44:56.113271 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:44:56.113172    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d4b9e97c-cfed-4de6-b410-4d8b1f3c98b5-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-636c4-predictor-5879d668f5-6hwtz\" (UID: \"d4b9e97c-cfed-4de6-b410-4d8b1f3c98b5\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-636c4-predictor-5879d668f5-6hwtz"
Sep 17 17:44:56.113596 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:44:56.113573    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d4b9e97c-cfed-4de6-b410-4d8b1f3c98b5-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-636c4-predictor-5879d668f5-6hwtz\" (UID: \"d4b9e97c-cfed-4de6-b410-4d8b1f3c98b5\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-636c4-predictor-5879d668f5-6hwtz"
Sep 17 17:44:56.113774 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:44:56.113754    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d4b9e97c-cfed-4de6-b410-4d8b1f3c98b5-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-636c4-predictor-5879d668f5-6hwtz\" (UID: \"d4b9e97c-cfed-4de6-b410-4d8b1f3c98b5\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-636c4-predictor-5879d668f5-6hwtz"
Sep 17 17:44:56.113828 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:44:56.113757    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d4b9e97c-cfed-4de6-b410-4d8b1f3c98b5-cabundle-cert\") pod \"isvc-sklearn-graph-raw-636c4-predictor-5879d668f5-6hwtz\" (UID: \"d4b9e97c-cfed-4de6-b410-4d8b1f3c98b5\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-636c4-predictor-5879d668f5-6hwtz"
Sep 17 17:44:56.115362 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:44:56.115341    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d4b9e97c-cfed-4de6-b410-4d8b1f3c98b5-proxy-tls\") pod \"isvc-sklearn-graph-raw-636c4-predictor-5879d668f5-6hwtz\" (UID: \"d4b9e97c-cfed-4de6-b410-4d8b1f3c98b5\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-636c4-predictor-5879d668f5-6hwtz"
Sep 17 17:44:56.121905 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:44:56.121883    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n85sp\" (UniqueName: \"kubernetes.io/projected/d4b9e97c-cfed-4de6-b410-4d8b1f3c98b5-kube-api-access-n85sp\") pod \"isvc-sklearn-graph-raw-636c4-predictor-5879d668f5-6hwtz\" (UID: \"d4b9e97c-cfed-4de6-b410-4d8b1f3c98b5\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-636c4-predictor-5879d668f5-6hwtz"
Sep 17 17:44:56.171828 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:44:56.171800    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j4ds_d3f553e8-b416-4d7d-bbef-991a0404b287/konnectivity-agent/0.log" limit=1024
Sep 17 17:44:56.200764 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:44:56.200741    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-636c4-predictor-5879d668f5-6hwtz"
Sep 17 17:44:56.350136 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:44:56.350060    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j4ds_d3f553e8-b416-4d7d-bbef-991a0404b287/konnectivity-agent/0.log" limit=1024
Sep 17 17:44:56.472083 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:44:56.471974    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-636c4-predictor-5879d668f5-6hwtz"]
Sep 17 17:44:56.474728 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:44:56.474701    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4b9e97c_cfed_4de6_b410_4d8b1f3c98b5.slice/crio-4080fe7bfc7516ff7a7feeaa0994b5e05baf834a97123c5ebcc8f3fb4685ad11 WatchSource:0}: Error finding container 4080fe7bfc7516ff7a7feeaa0994b5e05baf834a97123c5ebcc8f3fb4685ad11: Status 404 returned error can't find the container with id 4080fe7bfc7516ff7a7feeaa0994b5e05baf834a97123c5ebcc8f3fb4685ad11
Sep 17 17:44:56.726813 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:44:56.726710    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-636c4-predictor-5879d668f5-6hwtz" event={"ID":"d4b9e97c-cfed-4de6-b410-4d8b1f3c98b5","Type":"ContainerStarted","Data":"96fe4b63b5b395d1cef5bf4d1ad294be612aaaa3129adf252b2a0b67070b34dc"}
Sep 17 17:44:56.726813 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:44:56.726786    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-636c4-predictor-5879d668f5-6hwtz" event={"ID":"d4b9e97c-cfed-4de6-b410-4d8b1f3c98b5","Type":"ContainerStarted","Data":"4080fe7bfc7516ff7a7feeaa0994b5e05baf834a97123c5ebcc8f3fb4685ad11"}
Sep 17 17:44:56.953541 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:44:56.953498    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fb568536-6d13-451a-83b9-cb54fa1beb52" path="/var/lib/kubelet/pods/fb568536-6d13-451a-83b9-cb54fa1beb52/volumes"
Sep 17 17:45:00.743457 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:45:00.743421    2574 generic.go:356] "Generic (PLEG): container finished" podID="d4b9e97c-cfed-4de6-b410-4d8b1f3c98b5" containerID="96fe4b63b5b395d1cef5bf4d1ad294be612aaaa3129adf252b2a0b67070b34dc" exitCode=0
Sep 17 17:45:00.743856 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:45:00.743489    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-636c4-predictor-5879d668f5-6hwtz" event={"ID":"d4b9e97c-cfed-4de6-b410-4d8b1f3c98b5","Type":"ContainerDied","Data":"96fe4b63b5b395d1cef5bf4d1ad294be612aaaa3129adf252b2a0b67070b34dc"}
Sep 17 17:45:01.749790 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:45:01.749756    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-636c4-predictor-5879d668f5-6hwtz" event={"ID":"d4b9e97c-cfed-4de6-b410-4d8b1f3c98b5","Type":"ContainerStarted","Data":"5476615e20535b2f0663fec0978f5fd329c6a4bcfa8ed155c810de4735a1d088"}
Sep 17 17:45:01.750176 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:45:01.749818    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-636c4-predictor-5879d668f5-6hwtz" event={"ID":"d4b9e97c-cfed-4de6-b410-4d8b1f3c98b5","Type":"ContainerStarted","Data":"537ce729fd19bac01ea8c81c55495e431227d5a1d2ad11a62afc3d5742a88a34"}
Sep 17 17:45:01.750176 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:45:01.749996    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-636c4-predictor-5879d668f5-6hwtz"
Sep 17 17:45:01.771826 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:45:01.771769    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-636c4-predictor-5879d668f5-6hwtz" podStartSLOduration=3.125810496 podStartE2EDuration="6.771751069s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.645940573s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:44:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:45:01.771694455 +0000 UTC m=+656.960568161" watchObservedRunningTime="2026-09-17 17:45:01.771751069 +0000 UTC m=+656.960624775"
Sep 17 17:45:02.754210 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:45:02.754180    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-636c4-predictor-5879d668f5-6hwtz"
Sep 17 17:45:02.755384 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:45:02.755350    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-636c4-predictor-5879d668f5-6hwtz" podUID="d4b9e97c-cfed-4de6-b410-4d8b1f3c98b5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Sep 17 17:45:03.758331 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:45:03.758293    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-636c4-predictor-5879d668f5-6hwtz" podUID="d4b9e97c-cfed-4de6-b410-4d8b1f3c98b5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Sep 17 17:45:08.762869 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:45:08.762840    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-636c4-predictor-5879d668f5-6hwtz"
Sep 17 17:45:08.763364 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:45:08.763337    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-636c4-predictor-5879d668f5-6hwtz" podUID="d4b9e97c-cfed-4de6-b410-4d8b1f3c98b5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Sep 17 17:45:18.764158 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:45:18.764120    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-636c4-predictor-5879d668f5-6hwtz" podUID="d4b9e97c-cfed-4de6-b410-4d8b1f3c98b5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Sep 17 17:45:28.764219 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:45:28.764180    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-636c4-predictor-5879d668f5-6hwtz" podUID="d4b9e97c-cfed-4de6-b410-4d8b1f3c98b5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Sep 17 17:45:38.764006 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:45:38.763971    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-636c4-predictor-5879d668f5-6hwtz" podUID="d4b9e97c-cfed-4de6-b410-4d8b1f3c98b5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Sep 17 17:45:47.938233 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:45:47.938154    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j4ds_d3f553e8-b416-4d7d-bbef-991a0404b287/konnectivity-agent/0.log" limit=1024
Sep 17 17:45:48.763921 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:45:48.763884    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-636c4-predictor-5879d668f5-6hwtz" podUID="d4b9e97c-cfed-4de6-b410-4d8b1f3c98b5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Sep 17 17:45:58.763323 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:45:58.763273    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-636c4-predictor-5879d668f5-6hwtz" podUID="d4b9e97c-cfed-4de6-b410-4d8b1f3c98b5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Sep 17 17:46:08.763419 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:46:08.763376    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-636c4-predictor-5879d668f5-6hwtz" podUID="d4b9e97c-cfed-4de6-b410-4d8b1f3c98b5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Sep 17 17:46:18.764176 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:46:18.764145    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-636c4-predictor-5879d668f5-6hwtz"
Sep 17 17:46:20.773779 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:46:20.773747    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-786544c6c8-l84cg"]
Sep 17 17:46:20.774203 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:46:20.774024    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-786544c6c8-l84cg" podUID="2d0dd10a-cafd-491a-aac4-ebe2baf5c52e" containerName="kube-rbac-proxy" containerID="cri-o://41066f4f3f58233ed92a04276eaba3684581888362d7c24b602fe992a7987a94" gracePeriod=30
Sep 17 17:46:20.774203 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:46:20.773998    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-786544c6c8-l84cg" podUID="2d0dd10a-cafd-491a-aac4-ebe2baf5c52e" containerName="kserve-container" containerID="cri-o://21e5b44dce66d42d5a751116d64386fb51fdcd38307200bbd041078ea459006c" gracePeriod=30
Sep 17 17:46:20.815451 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:46:20.815425    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j4ds_d3f553e8-b416-4d7d-bbef-991a0404b287/konnectivity-agent/0.log" limit=1024
Sep 17 17:46:21.068609 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:46:21.068526    2574 generic.go:356] "Generic (PLEG): container finished" podID="2d0dd10a-cafd-491a-aac4-ebe2baf5c52e" containerID="41066f4f3f58233ed92a04276eaba3684581888362d7c24b602fe992a7987a94" exitCode=2
Sep 17 17:46:21.068609 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:46:21.068596    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-786544c6c8-l84cg" event={"ID":"2d0dd10a-cafd-491a-aac4-ebe2baf5c52e","Type":"ContainerDied","Data":"41066f4f3f58233ed92a04276eaba3684581888362d7c24b602fe992a7987a94"}
Sep 17 17:46:23.419718 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:46:23.419676    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-786544c6c8-l84cg" podUID="2d0dd10a-cafd-491a-aac4-ebe2baf5c52e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.25:8643/healthz\": dial tcp 10.134.0.25:8643: connect: connection refused"
Sep 17 17:46:23.424922 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:46:23.424894    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-786544c6c8-l84cg" podUID="2d0dd10a-cafd-491a-aac4-ebe2baf5c52e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Sep 17 17:46:24.246634 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:46:24.246600    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j4ds_d3f553e8-b416-4d7d-bbef-991a0404b287/konnectivity-agent/0.log" limit=1024
Sep 17 17:46:25.518483 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:46:25.518458    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-786544c6c8-l84cg"
Sep 17 17:46:25.578086 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:46:25.578061    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2d0dd10a-cafd-491a-aac4-ebe2baf5c52e-kube-rbac-proxy-sar-config\") pod \"2d0dd10a-cafd-491a-aac4-ebe2baf5c52e\" (UID: \"2d0dd10a-cafd-491a-aac4-ebe2baf5c52e\") "
Sep 17 17:46:25.578259 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:46:25.578096    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2d0dd10a-cafd-491a-aac4-ebe2baf5c52e-cabundle-cert\") pod \"2d0dd10a-cafd-491a-aac4-ebe2baf5c52e\" (UID: \"2d0dd10a-cafd-491a-aac4-ebe2baf5c52e\") "
Sep 17 17:46:25.578259 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:46:25.578119    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-knfkm\" (UniqueName: \"kubernetes.io/projected/2d0dd10a-cafd-491a-aac4-ebe2baf5c52e-kube-api-access-knfkm\") pod \"2d0dd10a-cafd-491a-aac4-ebe2baf5c52e\" (UID: \"2d0dd10a-cafd-491a-aac4-ebe2baf5c52e\") "
Sep 17 17:46:25.578259 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:46:25.578139    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2d0dd10a-cafd-491a-aac4-ebe2baf5c52e-kserve-provision-location\") pod \"2d0dd10a-cafd-491a-aac4-ebe2baf5c52e\" (UID: \"2d0dd10a-cafd-491a-aac4-ebe2baf5c52e\") "
Sep 17 17:46:25.578259 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:46:25.578189    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2d0dd10a-cafd-491a-aac4-ebe2baf5c52e-proxy-tls\") pod \"2d0dd10a-cafd-491a-aac4-ebe2baf5c52e\" (UID: \"2d0dd10a-cafd-491a-aac4-ebe2baf5c52e\") "
Sep 17 17:46:25.578522 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:46:25.578437    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d0dd10a-cafd-491a-aac4-ebe2baf5c52e-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "2d0dd10a-cafd-491a-aac4-ebe2baf5c52e" (UID: "2d0dd10a-cafd-491a-aac4-ebe2baf5c52e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:46:25.578560 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:46:25.578436    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d0dd10a-cafd-491a-aac4-ebe2baf5c52e-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "2d0dd10a-cafd-491a-aac4-ebe2baf5c52e" (UID: "2d0dd10a-cafd-491a-aac4-ebe2baf5c52e"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:46:25.578624 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:46:25.578584    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d0dd10a-cafd-491a-aac4-ebe2baf5c52e-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "2d0dd10a-cafd-491a-aac4-ebe2baf5c52e" (UID: "2d0dd10a-cafd-491a-aac4-ebe2baf5c52e"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:46:25.580115 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:46:25.580082    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d0dd10a-cafd-491a-aac4-ebe2baf5c52e-kube-api-access-knfkm" (OuterVolumeSpecName: "kube-api-access-knfkm") pod "2d0dd10a-cafd-491a-aac4-ebe2baf5c52e" (UID: "2d0dd10a-cafd-491a-aac4-ebe2baf5c52e"). InnerVolumeSpecName "kube-api-access-knfkm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:46:25.580216 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:46:25.580179    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2d0dd10a-cafd-491a-aac4-ebe2baf5c52e-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "2d0dd10a-cafd-491a-aac4-ebe2baf5c52e" (UID: "2d0dd10a-cafd-491a-aac4-ebe2baf5c52e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:46:25.679410 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:46:25.679347    2574 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2d0dd10a-cafd-491a-aac4-ebe2baf5c52e-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-96.ec2.internal\" DevicePath \"\""
Sep 17 17:46:25.679410 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:46:25.679372    2574 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2d0dd10a-cafd-491a-aac4-ebe2baf5c52e-cabundle-cert\") on node \"ip-10-0-134-96.ec2.internal\" DevicePath \"\""
Sep 17 17:46:25.679410 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:46:25.679381    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-knfkm\" (UniqueName: \"kubernetes.io/projected/2d0dd10a-cafd-491a-aac4-ebe2baf5c52e-kube-api-access-knfkm\") on node \"ip-10-0-134-96.ec2.internal\" DevicePath \"\""
Sep 17 17:46:25.679410 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:46:25.679390    2574 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2d0dd10a-cafd-491a-aac4-ebe2baf5c52e-kserve-provision-location\") on node \"ip-10-0-134-96.ec2.internal\" DevicePath \"\""
Sep 17 17:46:25.679410 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:46:25.679398    2574 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2d0dd10a-cafd-491a-aac4-ebe2baf5c52e-proxy-tls\") on node \"ip-10-0-134-96.ec2.internal\" DevicePath \"\""
Sep 17 17:46:25.980109 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:46:25.980021    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j4ds_d3f553e8-b416-4d7d-bbef-991a0404b287/konnectivity-agent/0.log" limit=1024
Sep 17 17:46:26.082987 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:46:26.082960    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j4ds_d3f553e8-b416-4d7d-bbef-991a0404b287/konnectivity-agent/0.log" limit=1024
Sep 17 17:46:26.089817 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:46:26.089787    2574 generic.go:356] "Generic (PLEG): container finished" podID="2d0dd10a-cafd-491a-aac4-ebe2baf5c52e" containerID="21e5b44dce66d42d5a751116d64386fb51fdcd38307200bbd041078ea459006c" exitCode=0
Sep 17 17:46:26.089975 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:46:26.089843    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-786544c6c8-l84cg"
Sep 17 17:46:26.089975 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:46:26.089864    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-786544c6c8-l84cg" event={"ID":"2d0dd10a-cafd-491a-aac4-ebe2baf5c52e","Type":"ContainerDied","Data":"21e5b44dce66d42d5a751116d64386fb51fdcd38307200bbd041078ea459006c"}
Sep 17 17:46:26.089975 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:46:26.089933    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-786544c6c8-l84cg" event={"ID":"2d0dd10a-cafd-491a-aac4-ebe2baf5c52e","Type":"ContainerDied","Data":"b32f01d7761944d4d23fdb84b27915c7dd96438d27faa752432f5523d8e7a174"}
Sep 17 17:46:26.089975 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:46:26.089969    2574 scope.go:117] "RemoveContainer" containerID="41066f4f3f58233ed92a04276eaba3684581888362d7c24b602fe992a7987a94"
Sep 17 17:46:26.098587 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:46:26.098568    2574 scope.go:117] "RemoveContainer" containerID="21e5b44dce66d42d5a751116d64386fb51fdcd38307200bbd041078ea459006c"
Sep 17 17:46:26.106007 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:46:26.105985    2574 scope.go:117] "RemoveContainer" containerID="3618723be650f18cf5f6b40cc0235738839f7bd6581582d6898fe3c9f5d085d1"
Sep 17 17:46:26.106361 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:46:26.106343    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-684df5466b-pqqwj"]
Sep 17 17:46:26.106733 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:46:26.106717    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2d0dd10a-cafd-491a-aac4-ebe2baf5c52e" containerName="storage-initializer"
Sep 17 17:46:26.106796 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:46:26.106735    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="2d0dd10a-cafd-491a-aac4-ebe2baf5c52e" containerName="storage-initializer"
Sep 17 17:46:26.106796 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:46:26.106751    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2d0dd10a-cafd-491a-aac4-ebe2baf5c52e" containerName="kube-rbac-proxy"
Sep 17 17:46:26.106796 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:46:26.106756    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="2d0dd10a-cafd-491a-aac4-ebe2baf5c52e" containerName="kube-rbac-proxy"
Sep 17 17:46:26.106796 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:46:26.106774    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2d0dd10a-cafd-491a-aac4-ebe2baf5c52e" containerName="kserve-container"
Sep 17 17:46:26.106796 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:46:26.106781    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="2d0dd10a-cafd-491a-aac4-ebe2baf5c52e" containerName="kserve-container"
Sep 17 17:46:26.106958 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:46:26.106833    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="2d0dd10a-cafd-491a-aac4-ebe2baf5c52e" containerName="kube-rbac-proxy"
Sep 17 17:46:26.106958 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:46:26.106841    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="2d0dd10a-cafd-491a-aac4-ebe2baf5c52e" containerName="kserve-container"
Sep 17 17:46:26.111402 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:46:26.111385    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-684df5466b-pqqwj"
Sep 17 17:46:26.113172 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:46:26.113154    2574 scope.go:117] "RemoveContainer" containerID="41066f4f3f58233ed92a04276eaba3684581888362d7c24b602fe992a7987a94"
Sep 17 17:46:26.113413 ip-10-0-134-96 kubenswrapper[2574]: E0917 17:46:26.113394    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"41066f4f3f58233ed92a04276eaba3684581888362d7c24b602fe992a7987a94\": container with ID starting with 41066f4f3f58233ed92a04276eaba3684581888362d7c24b602fe992a7987a94 not found: ID does not exist" containerID="41066f4f3f58233ed92a04276eaba3684581888362d7c24b602fe992a7987a94"
Sep 17 17:46:26.113466 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:46:26.113421    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41066f4f3f58233ed92a04276eaba3684581888362d7c24b602fe992a7987a94"} err="failed to get container status \"41066f4f3f58233ed92a04276eaba3684581888362d7c24b602fe992a7987a94\": rpc error: code = NotFound desc = could not find container \"41066f4f3f58233ed92a04276eaba3684581888362d7c24b602fe992a7987a94\": container with ID starting with 41066f4f3f58233ed92a04276eaba3684581888362d7c24b602fe992a7987a94 not found: ID does not exist"
Sep 17 17:46:26.113466 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:46:26.113439    2574 scope.go:117] "RemoveContainer" containerID="21e5b44dce66d42d5a751116d64386fb51fdcd38307200bbd041078ea459006c"
Sep 17 17:46:26.113753 ip-10-0-134-96 kubenswrapper[2574]: E0917 17:46:26.113733    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"21e5b44dce66d42d5a751116d64386fb51fdcd38307200bbd041078ea459006c\": container with ID starting with 21e5b44dce66d42d5a751116d64386fb51fdcd38307200bbd041078ea459006c not found: ID does not exist" containerID="21e5b44dce66d42d5a751116d64386fb51fdcd38307200bbd041078ea459006c"
Sep 17 17:46:26.113793 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:46:26.113760    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21e5b44dce66d42d5a751116d64386fb51fdcd38307200bbd041078ea459006c"} err="failed to get container status \"21e5b44dce66d42d5a751116d64386fb51fdcd38307200bbd041078ea459006c\": rpc error: code = NotFound desc = could not find container \"21e5b44dce66d42d5a751116d64386fb51fdcd38307200bbd041078ea459006c\": container with ID starting with 21e5b44dce66d42d5a751116d64386fb51fdcd38307200bbd041078ea459006c not found: ID does not exist"
Sep 17 17:46:26.113793 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:46:26.113776    2574 scope.go:117] "RemoveContainer" containerID="3618723be650f18cf5f6b40cc0235738839f7bd6581582d6898fe3c9f5d085d1"
Sep 17 17:46:26.113883 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:46:26.113801    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-predictor-serving-cert\""
Sep 17 17:46:26.113883 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:46:26.113801    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-kube-rbac-proxy-sar-config\""
Sep 17 17:46:26.114061 ip-10-0-134-96 kubenswrapper[2574]: E0917 17:46:26.114032    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3618723be650f18cf5f6b40cc0235738839f7bd6581582d6898fe3c9f5d085d1\": container with ID starting with 3618723be650f18cf5f6b40cc0235738839f7bd6581582d6898fe3c9f5d085d1 not found: ID does not exist" containerID="3618723be650f18cf5f6b40cc0235738839f7bd6581582d6898fe3c9f5d085d1"
Sep 17 17:46:26.114104 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:46:26.114068    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3618723be650f18cf5f6b40cc0235738839f7bd6581582d6898fe3c9f5d085d1"} err="failed to get container status \"3618723be650f18cf5f6b40cc0235738839f7bd6581582d6898fe3c9f5d085d1\": rpc error: code = NotFound desc = could not find container \"3618723be650f18cf5f6b40cc0235738839f7bd6581582d6898fe3c9f5d085d1\": container with ID starting with 3618723be650f18cf5f6b40cc0235738839f7bd6581582d6898fe3c9f5d085d1 not found: ID does not exist"
Sep 17 17:46:26.121844 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:46:26.121820    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-786544c6c8-l84cg"]
Sep 17 17:46:26.123098 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:46:26.123081    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-684df5466b-pqqwj"]
Sep 17 17:46:26.127472 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:46:26.127452    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-786544c6c8-l84cg"]
Sep 17 17:46:26.180512 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:46:26.180482    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j4ds_d3f553e8-b416-4d7d-bbef-991a0404b287/konnectivity-agent/0.log" limit=1024
Sep 17 17:46:26.184708 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:46:26.184684    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9e640c3f-798b-4734-9258-361fb5ccde21-kserve-provision-location\") pod \"isvc-canary-stop-predictor-684df5466b-pqqwj\" (UID: \"9e640c3f-798b-4734-9258-361fb5ccde21\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-684df5466b-pqqwj"
Sep 17 17:46:26.184826 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:46:26.184716    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9e640c3f-798b-4734-9258-361fb5ccde21-proxy-tls\") pod \"isvc-canary-stop-predictor-684df5466b-pqqwj\" (UID: \"9e640c3f-798b-4734-9258-361fb5ccde21\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-684df5466b-pqqwj"
Sep 17 17:46:26.184826 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:46:26.184738    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9e640c3f-798b-4734-9258-361fb5ccde21-cabundle-cert\") pod \"isvc-canary-stop-predictor-684df5466b-pqqwj\" (UID: \"9e640c3f-798b-4734-9258-361fb5ccde21\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-684df5466b-pqqwj"
Sep 17 17:46:26.184826 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:46:26.184764    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9e640c3f-798b-4734-9258-361fb5ccde21-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-684df5466b-pqqwj\" (UID: \"9e640c3f-798b-4734-9258-361fb5ccde21\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-684df5466b-pqqwj"
Sep 17 17:46:26.184826 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:46:26.184798    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qj4nd\" (UniqueName: \"kubernetes.io/projected/9e640c3f-798b-4734-9258-361fb5ccde21-kube-api-access-qj4nd\") pod \"isvc-canary-stop-predictor-684df5466b-pqqwj\" (UID: \"9e640c3f-798b-4734-9258-361fb5ccde21\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-684df5466b-pqqwj"
Sep 17 17:46:26.285636 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:46:26.285607    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9e640c3f-798b-4734-9258-361fb5ccde21-kserve-provision-location\") pod \"isvc-canary-stop-predictor-684df5466b-pqqwj\" (UID: \"9e640c3f-798b-4734-9258-361fb5ccde21\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-684df5466b-pqqwj"
Sep 17 17:46:26.285812 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:46:26.285642    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9e640c3f-798b-4734-9258-361fb5ccde21-proxy-tls\") pod \"isvc-canary-stop-predictor-684df5466b-pqqwj\" (UID: \"9e640c3f-798b-4734-9258-361fb5ccde21\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-684df5466b-pqqwj"
Sep 17 17:46:26.285812 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:46:26.285664    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9e640c3f-798b-4734-9258-361fb5ccde21-cabundle-cert\") pod \"isvc-canary-stop-predictor-684df5466b-pqqwj\" (UID: \"9e640c3f-798b-4734-9258-361fb5ccde21\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-684df5466b-pqqwj"
Sep 17 17:46:26.285812 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:46:26.285689    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9e640c3f-798b-4734-9258-361fb5ccde21-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-684df5466b-pqqwj\" (UID: \"9e640c3f-798b-4734-9258-361fb5ccde21\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-684df5466b-pqqwj"
Sep 17 17:46:26.285812 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:46:26.285713    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qj4nd\" (UniqueName: \"kubernetes.io/projected/9e640c3f-798b-4734-9258-361fb5ccde21-kube-api-access-qj4nd\") pod \"isvc-canary-stop-predictor-684df5466b-pqqwj\" (UID: \"9e640c3f-798b-4734-9258-361fb5ccde21\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-684df5466b-pqqwj"
Sep 17 17:46:26.286136 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:46:26.286114    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9e640c3f-798b-4734-9258-361fb5ccde21-kserve-provision-location\") pod \"isvc-canary-stop-predictor-684df5466b-pqqwj\" (UID: \"9e640c3f-798b-4734-9258-361fb5ccde21\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-684df5466b-pqqwj"
Sep 17 17:46:26.286458 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:46:26.286428    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9e640c3f-798b-4734-9258-361fb5ccde21-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-684df5466b-pqqwj\" (UID: \"9e640c3f-798b-4734-9258-361fb5ccde21\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-684df5466b-pqqwj"
Sep 17 17:46:26.286540 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:46:26.286482    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9e640c3f-798b-4734-9258-361fb5ccde21-cabundle-cert\") pod \"isvc-canary-stop-predictor-684df5466b-pqqwj\" (UID: \"9e640c3f-798b-4734-9258-361fb5ccde21\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-684df5466b-pqqwj"
Sep 17 17:46:26.286715 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:46:26.286692    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j4ds_d3f553e8-b416-4d7d-bbef-991a0404b287/konnectivity-agent/0.log" limit=1024
Sep 17 17:46:26.288310 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:46:26.288293    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9e640c3f-798b-4734-9258-361fb5ccde21-proxy-tls\") pod \"isvc-canary-stop-predictor-684df5466b-pqqwj\" (UID: \"9e640c3f-798b-4734-9258-361fb5ccde21\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-684df5466b-pqqwj"
Sep 17 17:46:26.297116 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:46:26.297092    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qj4nd\" (UniqueName: \"kubernetes.io/projected/9e640c3f-798b-4734-9258-361fb5ccde21-kube-api-access-qj4nd\") pod \"isvc-canary-stop-predictor-684df5466b-pqqwj\" (UID: \"9e640c3f-798b-4734-9258-361fb5ccde21\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-684df5466b-pqqwj"
Sep 17 17:46:26.420507 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:46:26.420478    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-684df5466b-pqqwj"
Sep 17 17:46:26.693327 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:46:26.693145    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-684df5466b-pqqwj"]
Sep 17 17:46:26.695420 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:46:26.695385    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e640c3f_798b_4734_9258_361fb5ccde21.slice/crio-2d660d0a97043bc512776bb4e99b4bc437d76450950ba061e7540585dfb024a6 WatchSource:0}: Error finding container 2d660d0a97043bc512776bb4e99b4bc437d76450950ba061e7540585dfb024a6: Status 404 returned error can't find the container with id 2d660d0a97043bc512776bb4e99b4bc437d76450950ba061e7540585dfb024a6
Sep 17 17:46:26.953592 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:46:26.953513    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2d0dd10a-cafd-491a-aac4-ebe2baf5c52e" path="/var/lib/kubelet/pods/2d0dd10a-cafd-491a-aac4-ebe2baf5c52e/volumes"
Sep 17 17:46:27.096356 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:46:27.096312    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-684df5466b-pqqwj" event={"ID":"9e640c3f-798b-4734-9258-361fb5ccde21","Type":"ContainerStarted","Data":"4b39febb3d72a6d5d2cd81b3593d7e281d7e3cf72b758070d99d83d3bce6e29b"}
Sep 17 17:46:27.096541 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:46:27.096382    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-684df5466b-pqqwj" event={"ID":"9e640c3f-798b-4734-9258-361fb5ccde21","Type":"ContainerStarted","Data":"2d660d0a97043bc512776bb4e99b4bc437d76450950ba061e7540585dfb024a6"}
Sep 17 17:46:31.113250 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:46:31.113218    2574 generic.go:356] "Generic (PLEG): container finished" podID="9e640c3f-798b-4734-9258-361fb5ccde21" containerID="4b39febb3d72a6d5d2cd81b3593d7e281d7e3cf72b758070d99d83d3bce6e29b" exitCode=0
Sep 17 17:46:31.113633 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:46:31.113287    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-684df5466b-pqqwj" event={"ID":"9e640c3f-798b-4734-9258-361fb5ccde21","Type":"ContainerDied","Data":"4b39febb3d72a6d5d2cd81b3593d7e281d7e3cf72b758070d99d83d3bce6e29b"}
Sep 17 17:46:32.119186 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:46:32.119145    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-684df5466b-pqqwj" event={"ID":"9e640c3f-798b-4734-9258-361fb5ccde21","Type":"ContainerStarted","Data":"1bca35fd19d49420b126bc6e0925496c298efc46970c71e33aa1f54788c954fe"}
Sep 17 17:46:32.119552 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:46:32.119215    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-684df5466b-pqqwj" event={"ID":"9e640c3f-798b-4734-9258-361fb5ccde21","Type":"ContainerStarted","Data":"48b8bc27cc857030e0ab785eb66cd927e6720cbb22273fa614e7b1e6f66a9a61"}
Sep 17 17:46:32.119552 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:46:32.119493    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-684df5466b-pqqwj"
Sep 17 17:46:32.119653 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:46:32.119633    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-684df5466b-pqqwj"
Sep 17 17:46:32.120927 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:46:32.120901    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-684df5466b-pqqwj" podUID="9e640c3f-798b-4734-9258-361fb5ccde21" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Sep 17 17:46:32.138741 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:46:32.138700    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-684df5466b-pqqwj" podStartSLOduration=2.402374893 podStartE2EDuration="6.138685555s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.736310662s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:46:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:46:32.138215987 +0000 UTC m=+747.327089706" watchObservedRunningTime="2026-09-17 17:46:32.138685555 +0000 UTC m=+747.327559313"
Sep 17 17:46:33.123065 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:46:33.123009    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-684df5466b-pqqwj" podUID="9e640c3f-798b-4734-9258-361fb5ccde21" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Sep 17 17:46:35.922061 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:46:35.922002    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j4ds_d3f553e8-b416-4d7d-bbef-991a0404b287/konnectivity-agent/0.log" limit=1024
Sep 17 17:46:36.270935 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:46:36.270905    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j4ds_d3f553e8-b416-4d7d-bbef-991a0404b287/konnectivity-agent/0.log" limit=1024
Sep 17 17:46:38.127428 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:46:38.127400    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-684df5466b-pqqwj"
Sep 17 17:46:38.127878 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:46:38.127855    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-684df5466b-pqqwj" podUID="9e640c3f-798b-4734-9258-361fb5ccde21" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Sep 17 17:46:45.967244 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:46:45.967212    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j4ds_d3f553e8-b416-4d7d-bbef-991a0404b287/konnectivity-agent/0.log" limit=1024
Sep 17 17:46:46.034714 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:46:46.034685    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-636c4-predictor-5879d668f5-6hwtz"]
Sep 17 17:46:46.035027 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:46:46.034993    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-636c4-predictor-5879d668f5-6hwtz" podUID="d4b9e97c-cfed-4de6-b410-4d8b1f3c98b5" containerName="kserve-container" containerID="cri-o://537ce729fd19bac01ea8c81c55495e431227d5a1d2ad11a62afc3d5742a88a34" gracePeriod=30
Sep 17 17:46:46.035027 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:46:46.034995    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-636c4-predictor-5879d668f5-6hwtz" podUID="d4b9e97c-cfed-4de6-b410-4d8b1f3c98b5" containerName="kube-rbac-proxy" containerID="cri-o://5476615e20535b2f0663fec0978f5fd329c6a4bcfa8ed155c810de4735a1d088" gracePeriod=30
Sep 17 17:46:46.081083 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:46:46.081033    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j4ds_d3f553e8-b416-4d7d-bbef-991a0404b287/konnectivity-agent/0.log" limit=1024
Sep 17 17:46:46.176944 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:46:46.176908    2574 generic.go:356] "Generic (PLEG): container finished" podID="d4b9e97c-cfed-4de6-b410-4d8b1f3c98b5" containerID="5476615e20535b2f0663fec0978f5fd329c6a4bcfa8ed155c810de4735a1d088" exitCode=2
Sep 17 17:46:46.177089 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:46:46.176974    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-636c4-predictor-5879d668f5-6hwtz" event={"ID":"d4b9e97c-cfed-4de6-b410-4d8b1f3c98b5","Type":"ContainerDied","Data":"5476615e20535b2f0663fec0978f5fd329c6a4bcfa8ed155c810de4735a1d088"}
Sep 17 17:46:46.185889 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:46:46.185862    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j4ds_d3f553e8-b416-4d7d-bbef-991a0404b287/konnectivity-agent/0.log" limit=1024
Sep 17 17:46:46.215093 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:46:46.215064    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-51535-predictor-574dd6d475-p6srw"]
Sep 17 17:46:46.218411 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:46:46.218375    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-51535-predictor-574dd6d475-p6srw"
Sep 17 17:46:46.220875 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:46:46.220858    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-hpa-51535-predictor-serving-cert\""
Sep 17 17:46:46.220981 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:46:46.220883    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-hpa-51535-kube-rbac-proxy-sar-config\""
Sep 17 17:46:46.230938 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:46:46.230918    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-51535-predictor-574dd6d475-p6srw"]
Sep 17 17:46:46.283988 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:46:46.283963    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j4ds_d3f553e8-b416-4d7d-bbef-991a0404b287/konnectivity-agent/0.log" limit=1024
Sep 17 17:46:46.336681 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:46:46.336659    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-51535-predictor-7bb954b865-jzlb4"]
Sep 17 17:46:46.339948 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:46:46.339932    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-51535-predictor-7bb954b865-jzlb4"
Sep 17 17:46:46.342234 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:46:46.342210    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-raw-hpa-51535-kube-rbac-proxy-sar-config\""
Sep 17 17:46:46.342320 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:46:46.342262    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-raw-hpa-51535-predictor-serving-cert\""
Sep 17 17:46:46.350990 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:46:46.350972    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-51535-predictor-7bb954b865-jzlb4"]
Sep 17 17:46:46.356490 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:46:46.356470    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5f644bf6-20a1-4abd-b7cf-23757cbb6f75-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-hpa-51535-predictor-574dd6d475-p6srw\" (UID: \"5f644bf6-20a1-4abd-b7cf-23757cbb6f75\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-51535-predictor-574dd6d475-p6srw"
Sep 17 17:46:46.356574 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:46:46.356503    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5f644bf6-20a1-4abd-b7cf-23757cbb6f75-proxy-tls\") pod \"isvc-sklearn-graph-raw-hpa-51535-predictor-574dd6d475-p6srw\" (UID: \"5f644bf6-20a1-4abd-b7cf-23757cbb6f75\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-51535-predictor-574dd6d475-p6srw"
Sep 17 17:46:46.356574 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:46:46.356534    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5f644bf6-20a1-4abd-b7cf-23757cbb6f75-cabundle-cert\") pod \"isvc-sklearn-graph-raw-hpa-51535-predictor-574dd6d475-p6srw\" (UID: \"5f644bf6-20a1-4abd-b7cf-23757cbb6f75\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-51535-predictor-574dd6d475-p6srw"
Sep 17 17:46:46.356644 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:46:46.356593    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5f644bf6-20a1-4abd-b7cf-23757cbb6f75-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-hpa-51535-predictor-574dd6d475-p6srw\" (UID: \"5f644bf6-20a1-4abd-b7cf-23757cbb6f75\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-51535-predictor-574dd6d475-p6srw"
Sep 17 17:46:46.356644 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:46:46.356621    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zs2wg\" (UniqueName: \"kubernetes.io/projected/5f644bf6-20a1-4abd-b7cf-23757cbb6f75-kube-api-access-zs2wg\") pod \"isvc-sklearn-graph-raw-hpa-51535-predictor-574dd6d475-p6srw\" (UID: \"5f644bf6-20a1-4abd-b7cf-23757cbb6f75\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-51535-predictor-574dd6d475-p6srw"
Sep 17 17:46:46.380916 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:46:46.380896    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j4ds_d3f553e8-b416-4d7d-bbef-991a0404b287/konnectivity-agent/0.log" limit=1024
Sep 17 17:46:46.457276 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:46:46.457253    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zs2wg\" (UniqueName: \"kubernetes.io/projected/5f644bf6-20a1-4abd-b7cf-23757cbb6f75-kube-api-access-zs2wg\") pod \"isvc-sklearn-graph-raw-hpa-51535-predictor-574dd6d475-p6srw\" (UID: \"5f644bf6-20a1-4abd-b7cf-23757cbb6f75\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-51535-predictor-574dd6d475-p6srw"
Sep 17 17:46:46.457371 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:46:46.457291    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b5d1c8b1-6c52-4981-925d-0d53b4416cad-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-hpa-51535-predictor-7bb954b865-jzlb4\" (UID: \"b5d1c8b1-6c52-4981-925d-0d53b4416cad\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-51535-predictor-7bb954b865-jzlb4"
Sep 17 17:46:46.457371 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:46:46.457320    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b5d1c8b1-6c52-4981-925d-0d53b4416cad-proxy-tls\") pod \"isvc-xgboost-graph-raw-hpa-51535-predictor-7bb954b865-jzlb4\" (UID: \"b5d1c8b1-6c52-4981-925d-0d53b4416cad\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-51535-predictor-7bb954b865-jzlb4"
Sep 17 17:46:46.457438 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:46:46.457407    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvnvl\" (UniqueName: \"kubernetes.io/projected/b5d1c8b1-6c52-4981-925d-0d53b4416cad-kube-api-access-jvnvl\") pod \"isvc-xgboost-graph-raw-hpa-51535-predictor-7bb954b865-jzlb4\" (UID: \"b5d1c8b1-6c52-4981-925d-0d53b4416cad\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-51535-predictor-7bb954b865-jzlb4"
Sep 17 17:46:46.457615 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:46:46.457453    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5f644bf6-20a1-4abd-b7cf-23757cbb6f75-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-hpa-51535-predictor-574dd6d475-p6srw\" (UID: \"5f644bf6-20a1-4abd-b7cf-23757cbb6f75\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-51535-predictor-574dd6d475-p6srw"
Sep 17 17:46:46.457615 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:46:46.457505    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5f644bf6-20a1-4abd-b7cf-23757cbb6f75-proxy-tls\") pod \"isvc-sklearn-graph-raw-hpa-51535-predictor-574dd6d475-p6srw\" (UID: \"5f644bf6-20a1-4abd-b7cf-23757cbb6f75\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-51535-predictor-574dd6d475-p6srw"
Sep 17 17:46:46.457615 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:46:46.457542    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b5d1c8b1-6c52-4981-925d-0d53b4416cad-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-hpa-51535-predictor-7bb954b865-jzlb4\" (UID: \"b5d1c8b1-6c52-4981-925d-0d53b4416cad\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-51535-predictor-7bb954b865-jzlb4"
Sep 17 17:46:46.457615 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:46:46.457573    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5f644bf6-20a1-4abd-b7cf-23757cbb6f75-cabundle-cert\") pod \"isvc-sklearn-graph-raw-hpa-51535-predictor-574dd6d475-p6srw\" (UID: \"5f644bf6-20a1-4abd-b7cf-23757cbb6f75\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-51535-predictor-574dd6d475-p6srw"
Sep 17 17:46:46.457840 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:46:46.457609    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b5d1c8b1-6c52-4981-925d-0d53b4416cad-cabundle-cert\") pod \"isvc-xgboost-graph-raw-hpa-51535-predictor-7bb954b865-jzlb4\" (UID: \"b5d1c8b1-6c52-4981-925d-0d53b4416cad\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-51535-predictor-7bb954b865-jzlb4"
Sep 17 17:46:46.457840 ip-10-0-134-96 kubenswrapper[2574]: E0917 17:46:46.457634    2574 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-51535-predictor-serving-cert: secret "isvc-sklearn-graph-raw-hpa-51535-predictor-serving-cert" not found
Sep 17 17:46:46.457840 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:46:46.457657    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5f644bf6-20a1-4abd-b7cf-23757cbb6f75-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-hpa-51535-predictor-574dd6d475-p6srw\" (UID: \"5f644bf6-20a1-4abd-b7cf-23757cbb6f75\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-51535-predictor-574dd6d475-p6srw"
Sep 17 17:46:46.457840 ip-10-0-134-96 kubenswrapper[2574]: E0917 17:46:46.457682    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5f644bf6-20a1-4abd-b7cf-23757cbb6f75-proxy-tls podName:5f644bf6-20a1-4abd-b7cf-23757cbb6f75 nodeName:}" failed. No retries permitted until 2026-09-17 17:46:46.957664108 +0000 UTC m=+762.146537793 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/5f644bf6-20a1-4abd-b7cf-23757cbb6f75-proxy-tls") pod "isvc-sklearn-graph-raw-hpa-51535-predictor-574dd6d475-p6srw" (UID: "5f644bf6-20a1-4abd-b7cf-23757cbb6f75") : secret "isvc-sklearn-graph-raw-hpa-51535-predictor-serving-cert" not found
Sep 17 17:46:46.457840 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:46:46.457805    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5f644bf6-20a1-4abd-b7cf-23757cbb6f75-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-hpa-51535-predictor-574dd6d475-p6srw\" (UID: \"5f644bf6-20a1-4abd-b7cf-23757cbb6f75\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-51535-predictor-574dd6d475-p6srw"
Sep 17 17:46:46.458338 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:46:46.458319    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5f644bf6-20a1-4abd-b7cf-23757cbb6f75-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-hpa-51535-predictor-574dd6d475-p6srw\" (UID: \"5f644bf6-20a1-4abd-b7cf-23757cbb6f75\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-51535-predictor-574dd6d475-p6srw"
Sep 17 17:46:46.458381 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:46:46.458319    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5f644bf6-20a1-4abd-b7cf-23757cbb6f75-cabundle-cert\") pod \"isvc-sklearn-graph-raw-hpa-51535-predictor-574dd6d475-p6srw\" (UID: \"5f644bf6-20a1-4abd-b7cf-23757cbb6f75\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-51535-predictor-574dd6d475-p6srw"
Sep 17 17:46:46.467531 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:46:46.467511    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zs2wg\" (UniqueName: \"kubernetes.io/projected/5f644bf6-20a1-4abd-b7cf-23757cbb6f75-kube-api-access-zs2wg\") pod \"isvc-sklearn-graph-raw-hpa-51535-predictor-574dd6d475-p6srw\" (UID: \"5f644bf6-20a1-4abd-b7cf-23757cbb6f75\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-51535-predictor-574dd6d475-p6srw"
Sep 17 17:46:46.558647 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:46:46.558612    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b5d1c8b1-6c52-4981-925d-0d53b4416cad-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-hpa-51535-predictor-7bb954b865-jzlb4\" (UID: \"b5d1c8b1-6c52-4981-925d-0d53b4416cad\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-51535-predictor-7bb954b865-jzlb4"
Sep 17 17:46:46.558776 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:46:46.558659    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b5d1c8b1-6c52-4981-925d-0d53b4416cad-proxy-tls\") pod \"isvc-xgboost-graph-raw-hpa-51535-predictor-7bb954b865-jzlb4\" (UID: \"b5d1c8b1-6c52-4981-925d-0d53b4416cad\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-51535-predictor-7bb954b865-jzlb4"
Sep 17 17:46:46.558776 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:46:46.558683    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvnvl\" (UniqueName: \"kubernetes.io/projected/b5d1c8b1-6c52-4981-925d-0d53b4416cad-kube-api-access-jvnvl\") pod \"isvc-xgboost-graph-raw-hpa-51535-predictor-7bb954b865-jzlb4\" (UID: \"b5d1c8b1-6c52-4981-925d-0d53b4416cad\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-51535-predictor-7bb954b865-jzlb4"
Sep 17 17:46:46.558776 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:46:46.558741    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b5d1c8b1-6c52-4981-925d-0d53b4416cad-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-hpa-51535-predictor-7bb954b865-jzlb4\" (UID: \"b5d1c8b1-6c52-4981-925d-0d53b4416cad\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-51535-predictor-7bb954b865-jzlb4"
Sep 17 17:46:46.558776 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:46:46.558770    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b5d1c8b1-6c52-4981-925d-0d53b4416cad-cabundle-cert\") pod \"isvc-xgboost-graph-raw-hpa-51535-predictor-7bb954b865-jzlb4\" (UID: \"b5d1c8b1-6c52-4981-925d-0d53b4416cad\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-51535-predictor-7bb954b865-jzlb4"
Sep 17 17:46:46.558943 ip-10-0-134-96 kubenswrapper[2574]: E0917 17:46:46.558776    2574 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-51535-predictor-serving-cert: secret "isvc-xgboost-graph-raw-hpa-51535-predictor-serving-cert" not found
Sep 17 17:46:46.558943 ip-10-0-134-96 kubenswrapper[2574]: E0917 17:46:46.558844    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b5d1c8b1-6c52-4981-925d-0d53b4416cad-proxy-tls podName:b5d1c8b1-6c52-4981-925d-0d53b4416cad nodeName:}" failed. No retries permitted until 2026-09-17 17:46:47.058826724 +0000 UTC m=+762.247700414 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/b5d1c8b1-6c52-4981-925d-0d53b4416cad-proxy-tls") pod "isvc-xgboost-graph-raw-hpa-51535-predictor-7bb954b865-jzlb4" (UID: "b5d1c8b1-6c52-4981-925d-0d53b4416cad") : secret "isvc-xgboost-graph-raw-hpa-51535-predictor-serving-cert" not found
Sep 17 17:46:46.559078 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:46:46.558996    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b5d1c8b1-6c52-4981-925d-0d53b4416cad-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-hpa-51535-predictor-7bb954b865-jzlb4\" (UID: \"b5d1c8b1-6c52-4981-925d-0d53b4416cad\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-51535-predictor-7bb954b865-jzlb4"
Sep 17 17:46:46.559325 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:46:46.559300    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b5d1c8b1-6c52-4981-925d-0d53b4416cad-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-hpa-51535-predictor-7bb954b865-jzlb4\" (UID: \"b5d1c8b1-6c52-4981-925d-0d53b4416cad\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-51535-predictor-7bb954b865-jzlb4"
Sep 17 17:46:46.559371 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:46:46.559337    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b5d1c8b1-6c52-4981-925d-0d53b4416cad-cabundle-cert\") pod \"isvc-xgboost-graph-raw-hpa-51535-predictor-7bb954b865-jzlb4\" (UID: \"b5d1c8b1-6c52-4981-925d-0d53b4416cad\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-51535-predictor-7bb954b865-jzlb4"
Sep 17 17:46:46.570386 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:46:46.570367    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvnvl\" (UniqueName: \"kubernetes.io/projected/b5d1c8b1-6c52-4981-925d-0d53b4416cad-kube-api-access-jvnvl\") pod \"isvc-xgboost-graph-raw-hpa-51535-predictor-7bb954b865-jzlb4\" (UID: \"b5d1c8b1-6c52-4981-925d-0d53b4416cad\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-51535-predictor-7bb954b865-jzlb4"
Sep 17 17:46:46.828288 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:46:46.828211    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j4ds_d3f553e8-b416-4d7d-bbef-991a0404b287/konnectivity-agent/0.log" limit=1024
Sep 17 17:46:46.962481 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:46:46.962448    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5f644bf6-20a1-4abd-b7cf-23757cbb6f75-proxy-tls\") pod \"isvc-sklearn-graph-raw-hpa-51535-predictor-574dd6d475-p6srw\" (UID: \"5f644bf6-20a1-4abd-b7cf-23757cbb6f75\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-51535-predictor-574dd6d475-p6srw"
Sep 17 17:46:46.964859 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:46:46.964838    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5f644bf6-20a1-4abd-b7cf-23757cbb6f75-proxy-tls\") pod \"isvc-sklearn-graph-raw-hpa-51535-predictor-574dd6d475-p6srw\" (UID: \"5f644bf6-20a1-4abd-b7cf-23757cbb6f75\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-51535-predictor-574dd6d475-p6srw"
Sep 17 17:46:46.969135 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:46:46.969101    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j4ds_d3f553e8-b416-4d7d-bbef-991a0404b287/konnectivity-agent/0.log" limit=1024
Sep 17 17:46:47.062864 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:46:47.062840    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b5d1c8b1-6c52-4981-925d-0d53b4416cad-proxy-tls\") pod \"isvc-xgboost-graph-raw-hpa-51535-predictor-7bb954b865-jzlb4\" (UID: \"b5d1c8b1-6c52-4981-925d-0d53b4416cad\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-51535-predictor-7bb954b865-jzlb4"
Sep 17 17:46:47.064997 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:46:47.064979    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b5d1c8b1-6c52-4981-925d-0d53b4416cad-proxy-tls\") pod \"isvc-xgboost-graph-raw-hpa-51535-predictor-7bb954b865-jzlb4\" (UID: \"b5d1c8b1-6c52-4981-925d-0d53b4416cad\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-51535-predictor-7bb954b865-jzlb4"
Sep 17 17:46:47.128137 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:46:47.128074    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-51535-predictor-574dd6d475-p6srw"
Sep 17 17:46:47.249661 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:46:47.249627    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-51535-predictor-7bb954b865-jzlb4"
Sep 17 17:46:47.403375 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:46:47.403352    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-51535-predictor-574dd6d475-p6srw"]
Sep 17 17:46:47.404770 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:46:47.404744    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f644bf6_20a1_4abd_b7cf_23757cbb6f75.slice/crio-a1590d60151105c08cf34ecabae27f538070cad06079fe09efe2b676bb56848d WatchSource:0}: Error finding container a1590d60151105c08cf34ecabae27f538070cad06079fe09efe2b676bb56848d: Status 404 returned error can't find the container with id a1590d60151105c08cf34ecabae27f538070cad06079fe09efe2b676bb56848d
Sep 17 17:46:47.522472 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:46:47.522437    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-51535-predictor-7bb954b865-jzlb4"]
Sep 17 17:46:47.525974 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:46:47.525948    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5d1c8b1_6c52_4981_925d_0d53b4416cad.slice/crio-a1a66d8b86ce28caa05f523984b824893dd72eeb295056a5e0e1f812b68260ce WatchSource:0}: Error finding container a1a66d8b86ce28caa05f523984b824893dd72eeb295056a5e0e1f812b68260ce: Status 404 returned error can't find the container with id a1a66d8b86ce28caa05f523984b824893dd72eeb295056a5e0e1f812b68260ce
Sep 17 17:46:48.128769 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:46:48.128721    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-684df5466b-pqqwj" podUID="9e640c3f-798b-4734-9258-361fb5ccde21" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Sep 17 17:46:48.189427 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:46:48.189393    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-51535-predictor-7bb954b865-jzlb4" event={"ID":"b5d1c8b1-6c52-4981-925d-0d53b4416cad","Type":"ContainerStarted","Data":"427d01bf4d7b1506317a01dbadf05887b4d829b978a3c3e5e7fc829bf4998a97"}
Sep 17 17:46:48.189620 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:46:48.189467    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-51535-predictor-7bb954b865-jzlb4" event={"ID":"b5d1c8b1-6c52-4981-925d-0d53b4416cad","Type":"ContainerStarted","Data":"a1a66d8b86ce28caa05f523984b824893dd72eeb295056a5e0e1f812b68260ce"}
Sep 17 17:46:48.190869 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:46:48.190838    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-51535-predictor-574dd6d475-p6srw" event={"ID":"5f644bf6-20a1-4abd-b7cf-23757cbb6f75","Type":"ContainerStarted","Data":"748a3655bda4902cefae13bf8949bd888fc6213bec66cd77ffc9f379f6a7eb9f"}
Sep 17 17:46:48.191009 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:46:48.190894    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-51535-predictor-574dd6d475-p6srw" event={"ID":"5f644bf6-20a1-4abd-b7cf-23757cbb6f75","Type":"ContainerStarted","Data":"a1590d60151105c08cf34ecabae27f538070cad06079fe09efe2b676bb56848d"}
Sep 17 17:46:48.759202 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:46:48.759163    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-636c4-predictor-5879d668f5-6hwtz" podUID="d4b9e97c-cfed-4de6-b410-4d8b1f3c98b5" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.26:8643/healthz\": dial tcp 10.134.0.26:8643: connect: connection refused"
Sep 17 17:46:48.763441 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:46:48.763403    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-636c4-predictor-5879d668f5-6hwtz" podUID="d4b9e97c-cfed-4de6-b410-4d8b1f3c98b5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Sep 17 17:46:50.474748 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:46:50.474721    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-636c4-predictor-5879d668f5-6hwtz"
Sep 17 17:46:50.591170 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:46:50.591097    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d4b9e97c-cfed-4de6-b410-4d8b1f3c98b5-proxy-tls\") pod \"d4b9e97c-cfed-4de6-b410-4d8b1f3c98b5\" (UID: \"d4b9e97c-cfed-4de6-b410-4d8b1f3c98b5\") "
Sep 17 17:46:50.591170 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:46:50.591138    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d4b9e97c-cfed-4de6-b410-4d8b1f3c98b5-kserve-provision-location\") pod \"d4b9e97c-cfed-4de6-b410-4d8b1f3c98b5\" (UID: \"d4b9e97c-cfed-4de6-b410-4d8b1f3c98b5\") "
Sep 17 17:46:50.591375 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:46:50.591177    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d4b9e97c-cfed-4de6-b410-4d8b1f3c98b5-kube-rbac-proxy-sar-config\") pod \"d4b9e97c-cfed-4de6-b410-4d8b1f3c98b5\" (UID: \"d4b9e97c-cfed-4de6-b410-4d8b1f3c98b5\") "
Sep 17 17:46:50.591375 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:46:50.591280    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d4b9e97c-cfed-4de6-b410-4d8b1f3c98b5-cabundle-cert\") pod \"d4b9e97c-cfed-4de6-b410-4d8b1f3c98b5\" (UID: \"d4b9e97c-cfed-4de6-b410-4d8b1f3c98b5\") "
Sep 17 17:46:50.591375 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:46:50.591323    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n85sp\" (UniqueName: \"kubernetes.io/projected/d4b9e97c-cfed-4de6-b410-4d8b1f3c98b5-kube-api-access-n85sp\") pod \"d4b9e97c-cfed-4de6-b410-4d8b1f3c98b5\" (UID: \"d4b9e97c-cfed-4de6-b410-4d8b1f3c98b5\") "
Sep 17 17:46:50.591564 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:46:50.591498    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4b9e97c-cfed-4de6-b410-4d8b1f3c98b5-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d4b9e97c-cfed-4de6-b410-4d8b1f3c98b5" (UID: "d4b9e97c-cfed-4de6-b410-4d8b1f3c98b5"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:46:50.591564 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:46:50.591548    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4b9e97c-cfed-4de6-b410-4d8b1f3c98b5-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "d4b9e97c-cfed-4de6-b410-4d8b1f3c98b5" (UID: "d4b9e97c-cfed-4de6-b410-4d8b1f3c98b5"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:46:50.591656 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:46:50.591563    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4b9e97c-cfed-4de6-b410-4d8b1f3c98b5-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "d4b9e97c-cfed-4de6-b410-4d8b1f3c98b5" (UID: "d4b9e97c-cfed-4de6-b410-4d8b1f3c98b5"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:46:50.591656 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:46:50.591597    2574 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d4b9e97c-cfed-4de6-b410-4d8b1f3c98b5-kserve-provision-location\") on node \"ip-10-0-134-96.ec2.internal\" DevicePath \"\""
Sep 17 17:46:50.592994 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:46:50.592963    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d4b9e97c-cfed-4de6-b410-4d8b1f3c98b5-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d4b9e97c-cfed-4de6-b410-4d8b1f3c98b5" (UID: "d4b9e97c-cfed-4de6-b410-4d8b1f3c98b5"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:46:50.593215 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:46:50.593195    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4b9e97c-cfed-4de6-b410-4d8b1f3c98b5-kube-api-access-n85sp" (OuterVolumeSpecName: "kube-api-access-n85sp") pod "d4b9e97c-cfed-4de6-b410-4d8b1f3c98b5" (UID: "d4b9e97c-cfed-4de6-b410-4d8b1f3c98b5"). InnerVolumeSpecName "kube-api-access-n85sp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:46:50.692761 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:46:50.692732    2574 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d4b9e97c-cfed-4de6-b410-4d8b1f3c98b5-proxy-tls\") on node \"ip-10-0-134-96.ec2.internal\" DevicePath \"\""
Sep 17 17:46:50.692761 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:46:50.692758    2574 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d4b9e97c-cfed-4de6-b410-4d8b1f3c98b5-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-96.ec2.internal\" DevicePath \"\""
Sep 17 17:46:50.692761 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:46:50.692767    2574 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d4b9e97c-cfed-4de6-b410-4d8b1f3c98b5-cabundle-cert\") on node \"ip-10-0-134-96.ec2.internal\" DevicePath \"\""
Sep 17 17:46:50.692940 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:46:50.692775    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n85sp\" (UniqueName: \"kubernetes.io/projected/d4b9e97c-cfed-4de6-b410-4d8b1f3c98b5-kube-api-access-n85sp\") on node \"ip-10-0-134-96.ec2.internal\" DevicePath \"\""
Sep 17 17:46:51.205826 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:46:51.205792    2574 generic.go:356] "Generic (PLEG): container finished" podID="d4b9e97c-cfed-4de6-b410-4d8b1f3c98b5" containerID="537ce729fd19bac01ea8c81c55495e431227d5a1d2ad11a62afc3d5742a88a34" exitCode=0
Sep 17 17:46:51.205983 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:46:51.205853    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-636c4-predictor-5879d668f5-6hwtz"
Sep 17 17:46:51.205983 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:46:51.205860    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-636c4-predictor-5879d668f5-6hwtz" event={"ID":"d4b9e97c-cfed-4de6-b410-4d8b1f3c98b5","Type":"ContainerDied","Data":"537ce729fd19bac01ea8c81c55495e431227d5a1d2ad11a62afc3d5742a88a34"}
Sep 17 17:46:51.205983 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:46:51.205928    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-636c4-predictor-5879d668f5-6hwtz" event={"ID":"d4b9e97c-cfed-4de6-b410-4d8b1f3c98b5","Type":"ContainerDied","Data":"4080fe7bfc7516ff7a7feeaa0994b5e05baf834a97123c5ebcc8f3fb4685ad11"}
Sep 17 17:46:51.205983 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:46:51.205965    2574 scope.go:117] "RemoveContainer" containerID="5476615e20535b2f0663fec0978f5fd329c6a4bcfa8ed155c810de4735a1d088"
Sep 17 17:46:51.213667 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:46:51.213647    2574 scope.go:117] "RemoveContainer" containerID="537ce729fd19bac01ea8c81c55495e431227d5a1d2ad11a62afc3d5742a88a34"
Sep 17 17:46:51.220744 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:46:51.220725    2574 scope.go:117] "RemoveContainer" containerID="96fe4b63b5b395d1cef5bf4d1ad294be612aaaa3129adf252b2a0b67070b34dc"
Sep 17 17:46:51.225840 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:46:51.225822    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-636c4-predictor-5879d668f5-6hwtz"]
Sep 17 17:46:51.228052 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:46:51.228028    2574 scope.go:117] "RemoveContainer" containerID="5476615e20535b2f0663fec0978f5fd329c6a4bcfa8ed155c810de4735a1d088"
Sep 17 17:46:51.228320 ip-10-0-134-96 kubenswrapper[2574]: E0917 17:46:51.228295    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5476615e20535b2f0663fec0978f5fd329c6a4bcfa8ed155c810de4735a1d088\": container with ID starting with 5476615e20535b2f0663fec0978f5fd329c6a4bcfa8ed155c810de4735a1d088 not found: ID does not exist" containerID="5476615e20535b2f0663fec0978f5fd329c6a4bcfa8ed155c810de4735a1d088"
Sep 17 17:46:51.228356 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:46:51.228331    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5476615e20535b2f0663fec0978f5fd329c6a4bcfa8ed155c810de4735a1d088"} err="failed to get container status \"5476615e20535b2f0663fec0978f5fd329c6a4bcfa8ed155c810de4735a1d088\": rpc error: code = NotFound desc = could not find container \"5476615e20535b2f0663fec0978f5fd329c6a4bcfa8ed155c810de4735a1d088\": container with ID starting with 5476615e20535b2f0663fec0978f5fd329c6a4bcfa8ed155c810de4735a1d088 not found: ID does not exist"
Sep 17 17:46:51.228356 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:46:51.228349    2574 scope.go:117] "RemoveContainer" containerID="537ce729fd19bac01ea8c81c55495e431227d5a1d2ad11a62afc3d5742a88a34"
Sep 17 17:46:51.228597 ip-10-0-134-96 kubenswrapper[2574]: E0917 17:46:51.228582    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"537ce729fd19bac01ea8c81c55495e431227d5a1d2ad11a62afc3d5742a88a34\": container with ID starting with 537ce729fd19bac01ea8c81c55495e431227d5a1d2ad11a62afc3d5742a88a34 not found: ID does not exist" containerID="537ce729fd19bac01ea8c81c55495e431227d5a1d2ad11a62afc3d5742a88a34"
Sep 17 17:46:51.228635 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:46:51.228602    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"537ce729fd19bac01ea8c81c55495e431227d5a1d2ad11a62afc3d5742a88a34"} err="failed to get container status \"537ce729fd19bac01ea8c81c55495e431227d5a1d2ad11a62afc3d5742a88a34\": rpc error: code = NotFound desc = could not find container \"537ce729fd19bac01ea8c81c55495e431227d5a1d2ad11a62afc3d5742a88a34\": container with ID starting with 537ce729fd19bac01ea8c81c55495e431227d5a1d2ad11a62afc3d5742a88a34 not found: ID does not exist"
Sep 17 17:46:51.228635 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:46:51.228616    2574 scope.go:117] "RemoveContainer" containerID="96fe4b63b5b395d1cef5bf4d1ad294be612aaaa3129adf252b2a0b67070b34dc"
Sep 17 17:46:51.228812 ip-10-0-134-96 kubenswrapper[2574]: E0917 17:46:51.228799    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"96fe4b63b5b395d1cef5bf4d1ad294be612aaaa3129adf252b2a0b67070b34dc\": container with ID starting with 96fe4b63b5b395d1cef5bf4d1ad294be612aaaa3129adf252b2a0b67070b34dc not found: ID does not exist" containerID="96fe4b63b5b395d1cef5bf4d1ad294be612aaaa3129adf252b2a0b67070b34dc"
Sep 17 17:46:51.228851 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:46:51.228816    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96fe4b63b5b395d1cef5bf4d1ad294be612aaaa3129adf252b2a0b67070b34dc"} err="failed to get container status \"96fe4b63b5b395d1cef5bf4d1ad294be612aaaa3129adf252b2a0b67070b34dc\": rpc error: code = NotFound desc = could not find container \"96fe4b63b5b395d1cef5bf4d1ad294be612aaaa3129adf252b2a0b67070b34dc\": container with ID starting with 96fe4b63b5b395d1cef5bf4d1ad294be612aaaa3129adf252b2a0b67070b34dc not found: ID does not exist"
Sep 17 17:46:51.234750 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:46:51.234728    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-636c4-predictor-5879d668f5-6hwtz"]
Sep 17 17:46:52.217698 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:46:52.217666    2574 generic.go:356] "Generic (PLEG): container finished" podID="b5d1c8b1-6c52-4981-925d-0d53b4416cad" containerID="427d01bf4d7b1506317a01dbadf05887b4d829b978a3c3e5e7fc829bf4998a97" exitCode=0
Sep 17 17:46:52.218141 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:46:52.217737    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-51535-predictor-7bb954b865-jzlb4" event={"ID":"b5d1c8b1-6c52-4981-925d-0d53b4416cad","Type":"ContainerDied","Data":"427d01bf4d7b1506317a01dbadf05887b4d829b978a3c3e5e7fc829bf4998a97"}
Sep 17 17:46:52.219019 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:46:52.219003    2574 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 17:46:52.219436 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:46:52.219414    2574 generic.go:356] "Generic (PLEG): container finished" podID="5f644bf6-20a1-4abd-b7cf-23757cbb6f75" containerID="748a3655bda4902cefae13bf8949bd888fc6213bec66cd77ffc9f379f6a7eb9f" exitCode=0
Sep 17 17:46:52.219521 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:46:52.219469    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-51535-predictor-574dd6d475-p6srw" event={"ID":"5f644bf6-20a1-4abd-b7cf-23757cbb6f75","Type":"ContainerDied","Data":"748a3655bda4902cefae13bf8949bd888fc6213bec66cd77ffc9f379f6a7eb9f"}
Sep 17 17:46:52.956698 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:46:52.956662    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d4b9e97c-cfed-4de6-b410-4d8b1f3c98b5" path="/var/lib/kubelet/pods/d4b9e97c-cfed-4de6-b410-4d8b1f3c98b5/volumes"
Sep 17 17:46:53.227670 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:46:53.227576    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-51535-predictor-574dd6d475-p6srw" event={"ID":"5f644bf6-20a1-4abd-b7cf-23757cbb6f75","Type":"ContainerStarted","Data":"c84a50fb176e9efb08c5bb2e4136f4fb0b157f15f0a5d9fc4695a991bf1d3e67"}
Sep 17 17:46:53.227670 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:46:53.227663    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-51535-predictor-574dd6d475-p6srw" event={"ID":"5f644bf6-20a1-4abd-b7cf-23757cbb6f75","Type":"ContainerStarted","Data":"de9069f2ba561ce5209b1f85f4dbca22bf523cef3fe256f677ed5e041144cbd5"}
Sep 17 17:46:53.228145 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:46:53.228019    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-51535-predictor-574dd6d475-p6srw"
Sep 17 17:46:53.228244 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:46:53.228227    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-51535-predictor-574dd6d475-p6srw"
Sep 17 17:46:53.229499 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:46:53.229468    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-51535-predictor-574dd6d475-p6srw" podUID="5f644bf6-20a1-4abd-b7cf-23757cbb6f75" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Sep 17 17:46:53.249317 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:46:53.249272    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-51535-predictor-574dd6d475-p6srw" podStartSLOduration=3.413311882 podStartE2EDuration="7.249255049s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.835943167s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:46:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:46:53.247091563 +0000 UTC m=+768.435965274" watchObservedRunningTime="2026-09-17 17:46:53.249255049 +0000 UTC m=+768.438128754"
Sep 17 17:46:54.233581 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:46:54.233522    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-51535-predictor-574dd6d475-p6srw" podUID="5f644bf6-20a1-4abd-b7cf-23757cbb6f75" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Sep 17 17:46:58.128226 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:46:58.128143    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-684df5466b-pqqwj" podUID="9e640c3f-798b-4734-9258-361fb5ccde21" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Sep 17 17:46:59.239150 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:46:59.239115    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-51535-predictor-574dd6d475-p6srw"
Sep 17 17:46:59.239779 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:46:59.239749    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-51535-predictor-574dd6d475-p6srw" podUID="5f644bf6-20a1-4abd-b7cf-23757cbb6f75" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Sep 17 17:47:08.128333 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:47:08.128286    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-684df5466b-pqqwj" podUID="9e640c3f-798b-4734-9258-361fb5ccde21" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Sep 17 17:47:09.240583 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:47:09.240537    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-51535-predictor-574dd6d475-p6srw" podUID="5f644bf6-20a1-4abd-b7cf-23757cbb6f75" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Sep 17 17:47:13.333387 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:47:13.333347    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-51535-predictor-7bb954b865-jzlb4" event={"ID":"b5d1c8b1-6c52-4981-925d-0d53b4416cad","Type":"ContainerStarted","Data":"b89efc67dfdf5d011bf5f4250b985bd75287254766ced757c929b9c12a305b15"}
Sep 17 17:47:13.333759 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:47:13.333440    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-51535-predictor-7bb954b865-jzlb4" event={"ID":"b5d1c8b1-6c52-4981-925d-0d53b4416cad","Type":"ContainerStarted","Data":"44ea00c134d5720335b050b1c812bd3613fc95cede151e9f556d2ca04fa76e9d"}
Sep 17 17:47:13.333802 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:47:13.333755    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-51535-predictor-7bb954b865-jzlb4"
Sep 17 17:47:13.358134 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:47:13.358086    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-51535-predictor-7bb954b865-jzlb4" podStartSLOduration=2.919771869 podStartE2EDuration="27.358069333s" totalImagesPullingTime="20.513258035s" totalInitContainerRuntime="3.925039429s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:46:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:47:13.355729254 +0000 UTC m=+788.544602961" watchObservedRunningTime="2026-09-17 17:47:13.358069333 +0000 UTC m=+788.546943037"
Sep 17 17:47:14.337270 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:47:14.337235    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-51535-predictor-7bb954b865-jzlb4"
Sep 17 17:47:14.338430 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:47:14.338396    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-51535-predictor-7bb954b865-jzlb4" podUID="b5d1c8b1-6c52-4981-925d-0d53b4416cad" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Sep 17 17:47:15.341653 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:47:15.341616    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-51535-predictor-7bb954b865-jzlb4" podUID="b5d1c8b1-6c52-4981-925d-0d53b4416cad" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Sep 17 17:47:17.034431 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:47:17.034398    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j4ds_d3f553e8-b416-4d7d-bbef-991a0404b287/konnectivity-agent/0.log" limit=1024
Sep 17 17:47:18.128798 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:47:18.128762    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-684df5466b-pqqwj" podUID="9e640c3f-798b-4734-9258-361fb5ccde21" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Sep 17 17:47:19.239960 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:47:19.239921    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-51535-predictor-574dd6d475-p6srw" podUID="5f644bf6-20a1-4abd-b7cf-23757cbb6f75" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Sep 17 17:47:20.346229 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:47:20.346197    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-51535-predictor-7bb954b865-jzlb4"
Sep 17 17:47:20.346712 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:47:20.346683    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-51535-predictor-7bb954b865-jzlb4" podUID="b5d1c8b1-6c52-4981-925d-0d53b4416cad" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Sep 17 17:47:28.128273 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:47:28.128232    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-684df5466b-pqqwj" podUID="9e640c3f-798b-4734-9258-361fb5ccde21" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Sep 17 17:47:29.239971 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:47:29.239934    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-51535-predictor-574dd6d475-p6srw" podUID="5f644bf6-20a1-4abd-b7cf-23757cbb6f75" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Sep 17 17:47:30.346808 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:47:30.346772    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-51535-predictor-7bb954b865-jzlb4" podUID="b5d1c8b1-6c52-4981-925d-0d53b4416cad" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Sep 17 17:47:38.128710 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:47:38.128668    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-684df5466b-pqqwj" podUID="9e640c3f-798b-4734-9258-361fb5ccde21" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Sep 17 17:47:39.240304 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:47:39.240261    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-51535-predictor-574dd6d475-p6srw" podUID="5f644bf6-20a1-4abd-b7cf-23757cbb6f75" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Sep 17 17:47:40.347358 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:47:40.347322    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-51535-predictor-7bb954b865-jzlb4" podUID="b5d1c8b1-6c52-4981-925d-0d53b4416cad" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Sep 17 17:47:41.949394 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:47:41.949353    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-684df5466b-pqqwj" podUID="9e640c3f-798b-4734-9258-361fb5ccde21" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Sep 17 17:47:49.239804 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:47:49.239766    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-51535-predictor-574dd6d475-p6srw" podUID="5f644bf6-20a1-4abd-b7cf-23757cbb6f75" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Sep 17 17:47:50.346722 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:47:50.346683    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-51535-predictor-7bb954b865-jzlb4" podUID="b5d1c8b1-6c52-4981-925d-0d53b4416cad" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Sep 17 17:47:51.953276 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:47:51.950287    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-684df5466b-pqqwj"
Sep 17 17:47:56.008490 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:47:56.008460    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j4ds_d3f553e8-b416-4d7d-bbef-991a0404b287/konnectivity-agent/0.log" limit=1024
Sep 17 17:47:56.011483 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:47:56.011459    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-684df5466b-pqqwj"]
Sep 17 17:47:56.011693 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:47:56.011652    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-684df5466b-pqqwj" podUID="9e640c3f-798b-4734-9258-361fb5ccde21" containerName="kserve-container" containerID="cri-o://48b8bc27cc857030e0ab785eb66cd927e6720cbb22273fa614e7b1e6f66a9a61" gracePeriod=30
Sep 17 17:47:56.011693 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:47:56.011678    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-684df5466b-pqqwj" podUID="9e640c3f-798b-4734-9258-361fb5ccde21" containerName="kube-rbac-proxy" containerID="cri-o://1bca35fd19d49420b126bc6e0925496c298efc46970c71e33aa1f54788c954fe" gracePeriod=30
Sep 17 17:47:56.107962 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:47:56.107931    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j4ds_d3f553e8-b416-4d7d-bbef-991a0404b287/konnectivity-agent/0.log" limit=1024
Sep 17 17:47:56.511371 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:47:56.511342    2574 generic.go:356] "Generic (PLEG): container finished" podID="9e640c3f-798b-4734-9258-361fb5ccde21" containerID="1bca35fd19d49420b126bc6e0925496c298efc46970c71e33aa1f54788c954fe" exitCode=2
Sep 17 17:47:56.511529 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:47:56.511417    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-684df5466b-pqqwj" event={"ID":"9e640c3f-798b-4734-9258-361fb5ccde21","Type":"ContainerDied","Data":"1bca35fd19d49420b126bc6e0925496c298efc46970c71e33aa1f54788c954fe"}
Sep 17 17:47:58.123385 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:47:58.123345    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-684df5466b-pqqwj" podUID="9e640c3f-798b-4734-9258-361fb5ccde21" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.27:8643/healthz\": dial tcp 10.134.0.27:8643: connect: connection refused"
Sep 17 17:47:59.240208 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:47:59.240169    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-51535-predictor-574dd6d475-p6srw" podUID="5f644bf6-20a1-4abd-b7cf-23757cbb6f75" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Sep 17 17:48:00.346663 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:48:00.346625    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-51535-predictor-7bb954b865-jzlb4" podUID="b5d1c8b1-6c52-4981-925d-0d53b4416cad" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Sep 17 17:48:00.531502 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:48:00.531464    2574 generic.go:356] "Generic (PLEG): container finished" podID="9e640c3f-798b-4734-9258-361fb5ccde21" containerID="48b8bc27cc857030e0ab785eb66cd927e6720cbb22273fa614e7b1e6f66a9a61" exitCode=0
Sep 17 17:48:00.531643 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:48:00.531535    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-684df5466b-pqqwj" event={"ID":"9e640c3f-798b-4734-9258-361fb5ccde21","Type":"ContainerDied","Data":"48b8bc27cc857030e0ab785eb66cd927e6720cbb22273fa614e7b1e6f66a9a61"}
Sep 17 17:48:00.548295 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:48:00.548273    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-684df5466b-pqqwj"
Sep 17 17:48:00.585784 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:48:00.585752    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9e640c3f-798b-4734-9258-361fb5ccde21-kube-rbac-proxy-sar-config\") pod \"9e640c3f-798b-4734-9258-361fb5ccde21\" (UID: \"9e640c3f-798b-4734-9258-361fb5ccde21\") "
Sep 17 17:48:00.585932 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:48:00.585808    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qj4nd\" (UniqueName: \"kubernetes.io/projected/9e640c3f-798b-4734-9258-361fb5ccde21-kube-api-access-qj4nd\") pod \"9e640c3f-798b-4734-9258-361fb5ccde21\" (UID: \"9e640c3f-798b-4734-9258-361fb5ccde21\") "
Sep 17 17:48:00.585932 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:48:00.585909    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9e640c3f-798b-4734-9258-361fb5ccde21-kserve-provision-location\") pod \"9e640c3f-798b-4734-9258-361fb5ccde21\" (UID: \"9e640c3f-798b-4734-9258-361fb5ccde21\") "
Sep 17 17:48:00.586013 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:48:00.585955    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9e640c3f-798b-4734-9258-361fb5ccde21-cabundle-cert\") pod \"9e640c3f-798b-4734-9258-361fb5ccde21\" (UID: \"9e640c3f-798b-4734-9258-361fb5ccde21\") "
Sep 17 17:48:00.586013 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:48:00.585994    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9e640c3f-798b-4734-9258-361fb5ccde21-proxy-tls\") pod \"9e640c3f-798b-4734-9258-361fb5ccde21\" (UID: \"9e640c3f-798b-4734-9258-361fb5ccde21\") "
Sep 17 17:48:00.586154 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:48:00.586119    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e640c3f-798b-4734-9258-361fb5ccde21-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "9e640c3f-798b-4734-9258-361fb5ccde21" (UID: "9e640c3f-798b-4734-9258-361fb5ccde21"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:48:00.586247 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:48:00.586222    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e640c3f-798b-4734-9258-361fb5ccde21-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "9e640c3f-798b-4734-9258-361fb5ccde21" (UID: "9e640c3f-798b-4734-9258-361fb5ccde21"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:48:00.586307 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:48:00.586270    2574 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9e640c3f-798b-4734-9258-361fb5ccde21-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-96.ec2.internal\" DevicePath \"\""
Sep 17 17:48:00.586359 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:48:00.586335    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e640c3f-798b-4734-9258-361fb5ccde21-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "9e640c3f-798b-4734-9258-361fb5ccde21" (UID: "9e640c3f-798b-4734-9258-361fb5ccde21"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:48:00.587871 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:48:00.587848    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e640c3f-798b-4734-9258-361fb5ccde21-kube-api-access-qj4nd" (OuterVolumeSpecName: "kube-api-access-qj4nd") pod "9e640c3f-798b-4734-9258-361fb5ccde21" (UID: "9e640c3f-798b-4734-9258-361fb5ccde21"). InnerVolumeSpecName "kube-api-access-qj4nd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:48:00.587967 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:48:00.587909    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9e640c3f-798b-4734-9258-361fb5ccde21-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "9e640c3f-798b-4734-9258-361fb5ccde21" (UID: "9e640c3f-798b-4734-9258-361fb5ccde21"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:48:00.687190 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:48:00.687117    2574 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9e640c3f-798b-4734-9258-361fb5ccde21-cabundle-cert\") on node \"ip-10-0-134-96.ec2.internal\" DevicePath \"\""
Sep 17 17:48:00.687190 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:48:00.687142    2574 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9e640c3f-798b-4734-9258-361fb5ccde21-proxy-tls\") on node \"ip-10-0-134-96.ec2.internal\" DevicePath \"\""
Sep 17 17:48:00.687190 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:48:00.687152    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qj4nd\" (UniqueName: \"kubernetes.io/projected/9e640c3f-798b-4734-9258-361fb5ccde21-kube-api-access-qj4nd\") on node \"ip-10-0-134-96.ec2.internal\" DevicePath \"\""
Sep 17 17:48:00.687190 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:48:00.687162    2574 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9e640c3f-798b-4734-9258-361fb5ccde21-kserve-provision-location\") on node \"ip-10-0-134-96.ec2.internal\" DevicePath \"\""
Sep 17 17:48:01.537247 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:48:01.537218    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-684df5466b-pqqwj"
Sep 17 17:48:01.537247 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:48:01.537244    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-684df5466b-pqqwj" event={"ID":"9e640c3f-798b-4734-9258-361fb5ccde21","Type":"ContainerDied","Data":"2d660d0a97043bc512776bb4e99b4bc437d76450950ba061e7540585dfb024a6"}
Sep 17 17:48:01.537781 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:48:01.537304    2574 scope.go:117] "RemoveContainer" containerID="1bca35fd19d49420b126bc6e0925496c298efc46970c71e33aa1f54788c954fe"
Sep 17 17:48:01.545170 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:48:01.545150    2574 scope.go:117] "RemoveContainer" containerID="48b8bc27cc857030e0ab785eb66cd927e6720cbb22273fa614e7b1e6f66a9a61"
Sep 17 17:48:01.552258 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:48:01.552233    2574 scope.go:117] "RemoveContainer" containerID="4b39febb3d72a6d5d2cd81b3593d7e281d7e3cf72b758070d99d83d3bce6e29b"
Sep 17 17:48:01.560465 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:48:01.560446    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-684df5466b-pqqwj"]
Sep 17 17:48:01.564761 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:48:01.564742    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-684df5466b-pqqwj"]
Sep 17 17:48:02.953472 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:48:02.953437    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9e640c3f-798b-4734-9258-361fb5ccde21" path="/var/lib/kubelet/pods/9e640c3f-798b-4734-9258-361fb5ccde21/volumes"
Sep 17 17:48:06.248169 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:48:06.248139    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j4ds_d3f553e8-b416-4d7d-bbef-991a0404b287/konnectivity-agent/0.log" limit=1024
Sep 17 17:48:06.572550 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:48:06.572455    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j4ds_d3f553e8-b416-4d7d-bbef-991a0404b287/konnectivity-agent/0.log" limit=1024
Sep 17 17:48:09.241205 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:48:09.241173    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-51535-predictor-574dd6d475-p6srw"
Sep 17 17:48:10.347227 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:48:10.347189    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-51535-predictor-7bb954b865-jzlb4" podUID="b5d1c8b1-6c52-4981-925d-0d53b4416cad" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Sep 17 17:48:20.347247 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:48:20.347212    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-51535-predictor-7bb954b865-jzlb4"
Sep 17 17:48:26.315427 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:48:26.315394    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j4ds_d3f553e8-b416-4d7d-bbef-991a0404b287/konnectivity-agent/0.log" limit=1024
Sep 17 17:48:26.528774 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:48:26.528743    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j4ds_d3f553e8-b416-4d7d-bbef-991a0404b287/konnectivity-agent/0.log" limit=1024
Sep 17 17:48:36.422281 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:48:36.422242    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j4ds_d3f553e8-b416-4d7d-bbef-991a0404b287/konnectivity-agent/0.log" limit=1024
Sep 17 17:48:36.465922 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:48:36.465882    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-51535-predictor-574dd6d475-p6srw"]
Sep 17 17:48:36.466181 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:48:36.466156    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-51535-predictor-574dd6d475-p6srw" podUID="5f644bf6-20a1-4abd-b7cf-23757cbb6f75" containerName="kserve-container" containerID="cri-o://de9069f2ba561ce5209b1f85f4dbca22bf523cef3fe256f677ed5e041144cbd5" gracePeriod=30
Sep 17 17:48:36.466323 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:48:36.466230    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-51535-predictor-574dd6d475-p6srw" podUID="5f644bf6-20a1-4abd-b7cf-23757cbb6f75" containerName="kube-rbac-proxy" containerID="cri-o://c84a50fb176e9efb08c5bb2e4136f4fb0b157f15f0a5d9fc4695a991bf1d3e67" gracePeriod=30
Sep 17 17:48:36.485101 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:48:36.485078    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-51535-predictor-7bb954b865-jzlb4"]
Sep 17 17:48:36.485317 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:48:36.485298    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-51535-predictor-7bb954b865-jzlb4" podUID="b5d1c8b1-6c52-4981-925d-0d53b4416cad" containerName="kserve-container" containerID="cri-o://44ea00c134d5720335b050b1c812bd3613fc95cede151e9f556d2ca04fa76e9d" gracePeriod=30
Sep 17 17:48:36.485387 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:48:36.485365    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-51535-predictor-7bb954b865-jzlb4" podUID="b5d1c8b1-6c52-4981-925d-0d53b4416cad" containerName="kube-rbac-proxy" containerID="cri-o://b89efc67dfdf5d011bf5f4250b985bd75287254766ced757c929b9c12a305b15" gracePeriod=30
Sep 17 17:48:36.679211 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:48:36.679122    2574 generic.go:356] "Generic (PLEG): container finished" podID="b5d1c8b1-6c52-4981-925d-0d53b4416cad" containerID="b89efc67dfdf5d011bf5f4250b985bd75287254766ced757c929b9c12a305b15" exitCode=2
Sep 17 17:48:36.679343 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:48:36.679206    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-51535-predictor-7bb954b865-jzlb4" event={"ID":"b5d1c8b1-6c52-4981-925d-0d53b4416cad","Type":"ContainerDied","Data":"b89efc67dfdf5d011bf5f4250b985bd75287254766ced757c929b9c12a305b15"}
Sep 17 17:48:36.681451 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:48:36.681424    2574 generic.go:356] "Generic (PLEG): container finished" podID="5f644bf6-20a1-4abd-b7cf-23757cbb6f75" containerID="c84a50fb176e9efb08c5bb2e4136f4fb0b157f15f0a5d9fc4695a991bf1d3e67" exitCode=2
Sep 17 17:48:36.681625 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:48:36.681475    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-51535-predictor-574dd6d475-p6srw" event={"ID":"5f644bf6-20a1-4abd-b7cf-23757cbb6f75","Type":"ContainerDied","Data":"c84a50fb176e9efb08c5bb2e4136f4fb0b157f15f0a5d9fc4695a991bf1d3e67"}
Sep 17 17:48:36.725386 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:48:36.725354    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j4ds_d3f553e8-b416-4d7d-bbef-991a0404b287/konnectivity-agent/0.log" limit=1024
Sep 17 17:48:36.901306 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:48:36.901271    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j4ds_d3f553e8-b416-4d7d-bbef-991a0404b287/konnectivity-agent/0.log" limit=1024
Sep 17 17:48:39.234246 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:48:39.234207    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-51535-predictor-574dd6d475-p6srw" podUID="5f644bf6-20a1-4abd-b7cf-23757cbb6f75" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.28:8643/healthz\": dial tcp 10.134.0.28:8643: connect: connection refused"
Sep 17 17:48:39.239898 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:48:39.239874    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-51535-predictor-574dd6d475-p6srw" podUID="5f644bf6-20a1-4abd-b7cf-23757cbb6f75" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Sep 17 17:48:40.342733 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:48:40.342690    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-51535-predictor-7bb954b865-jzlb4" podUID="b5d1c8b1-6c52-4981-925d-0d53b4416cad" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.29:8643/healthz\": dial tcp 10.134.0.29:8643: connect: connection refused"
Sep 17 17:48:40.347079 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:48:40.347032    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-51535-predictor-7bb954b865-jzlb4" podUID="b5d1c8b1-6c52-4981-925d-0d53b4416cad" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Sep 17 17:48:40.535201 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:48:40.535177    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-51535-predictor-7bb954b865-jzlb4"
Sep 17 17:48:40.621055 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:48:40.620959    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b5d1c8b1-6c52-4981-925d-0d53b4416cad-cabundle-cert\") pod \"b5d1c8b1-6c52-4981-925d-0d53b4416cad\" (UID: \"b5d1c8b1-6c52-4981-925d-0d53b4416cad\") "
Sep 17 17:48:40.621055 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:48:40.621011    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b5d1c8b1-6c52-4981-925d-0d53b4416cad-kube-rbac-proxy-sar-config\") pod \"b5d1c8b1-6c52-4981-925d-0d53b4416cad\" (UID: \"b5d1c8b1-6c52-4981-925d-0d53b4416cad\") "
Sep 17 17:48:40.621318 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:48:40.621296    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5d1c8b1-6c52-4981-925d-0d53b4416cad-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "b5d1c8b1-6c52-4981-925d-0d53b4416cad" (UID: "b5d1c8b1-6c52-4981-925d-0d53b4416cad"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:48:40.621366 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:48:40.621348    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5d1c8b1-6c52-4981-925d-0d53b4416cad-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "b5d1c8b1-6c52-4981-925d-0d53b4416cad" (UID: "b5d1c8b1-6c52-4981-925d-0d53b4416cad"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:48:40.698112 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:48:40.698081    2574 generic.go:356] "Generic (PLEG): container finished" podID="b5d1c8b1-6c52-4981-925d-0d53b4416cad" containerID="44ea00c134d5720335b050b1c812bd3613fc95cede151e9f556d2ca04fa76e9d" exitCode=0
Sep 17 17:48:40.698277 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:48:40.698129    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-51535-predictor-7bb954b865-jzlb4" event={"ID":"b5d1c8b1-6c52-4981-925d-0d53b4416cad","Type":"ContainerDied","Data":"44ea00c134d5720335b050b1c812bd3613fc95cede151e9f556d2ca04fa76e9d"}
Sep 17 17:48:40.698277 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:48:40.698141    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-51535-predictor-7bb954b865-jzlb4"
Sep 17 17:48:40.698277 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:48:40.698184    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-51535-predictor-7bb954b865-jzlb4" event={"ID":"b5d1c8b1-6c52-4981-925d-0d53b4416cad","Type":"ContainerDied","Data":"a1a66d8b86ce28caa05f523984b824893dd72eeb295056a5e0e1f812b68260ce"}
Sep 17 17:48:40.698277 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:48:40.698212    2574 scope.go:117] "RemoveContainer" containerID="b89efc67dfdf5d011bf5f4250b985bd75287254766ced757c929b9c12a305b15"
Sep 17 17:48:40.705963 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:48:40.705951    2574 scope.go:117] "RemoveContainer" containerID="44ea00c134d5720335b050b1c812bd3613fc95cede151e9f556d2ca04fa76e9d"
Sep 17 17:48:40.719291 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:48:40.719261    2574 scope.go:117] "RemoveContainer" containerID="427d01bf4d7b1506317a01dbadf05887b4d829b978a3c3e5e7fc829bf4998a97"
Sep 17 17:48:40.721483 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:48:40.721463    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b5d1c8b1-6c52-4981-925d-0d53b4416cad-kserve-provision-location\") pod \"b5d1c8b1-6c52-4981-925d-0d53b4416cad\" (UID: \"b5d1c8b1-6c52-4981-925d-0d53b4416cad\") "
Sep 17 17:48:40.721578 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:48:40.721499    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jvnvl\" (UniqueName: \"kubernetes.io/projected/b5d1c8b1-6c52-4981-925d-0d53b4416cad-kube-api-access-jvnvl\") pod \"b5d1c8b1-6c52-4981-925d-0d53b4416cad\" (UID: \"b5d1c8b1-6c52-4981-925d-0d53b4416cad\") "
Sep 17 17:48:40.721578 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:48:40.721535    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b5d1c8b1-6c52-4981-925d-0d53b4416cad-proxy-tls\") pod \"b5d1c8b1-6c52-4981-925d-0d53b4416cad\" (UID: \"b5d1c8b1-6c52-4981-925d-0d53b4416cad\") "
Sep 17 17:48:40.721816 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:48:40.721790    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5d1c8b1-6c52-4981-925d-0d53b4416cad-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "b5d1c8b1-6c52-4981-925d-0d53b4416cad" (UID: "b5d1c8b1-6c52-4981-925d-0d53b4416cad"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:48:40.721883 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:48:40.721828    2574 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b5d1c8b1-6c52-4981-925d-0d53b4416cad-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-96.ec2.internal\" DevicePath \"\""
Sep 17 17:48:40.721883 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:48:40.721848    2574 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b5d1c8b1-6c52-4981-925d-0d53b4416cad-cabundle-cert\") on node \"ip-10-0-134-96.ec2.internal\" DevicePath \"\""
Sep 17 17:48:40.723497 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:48:40.723473    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b5d1c8b1-6c52-4981-925d-0d53b4416cad-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "b5d1c8b1-6c52-4981-925d-0d53b4416cad" (UID: "b5d1c8b1-6c52-4981-925d-0d53b4416cad"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:48:40.723940 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:48:40.723921    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5d1c8b1-6c52-4981-925d-0d53b4416cad-kube-api-access-jvnvl" (OuterVolumeSpecName: "kube-api-access-jvnvl") pod "b5d1c8b1-6c52-4981-925d-0d53b4416cad" (UID: "b5d1c8b1-6c52-4981-925d-0d53b4416cad"). InnerVolumeSpecName "kube-api-access-jvnvl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:48:40.734411 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:48:40.734379    2574 scope.go:117] "RemoveContainer" containerID="b89efc67dfdf5d011bf5f4250b985bd75287254766ced757c929b9c12a305b15"
Sep 17 17:48:40.734676 ip-10-0-134-96 kubenswrapper[2574]: E0917 17:48:40.734662    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b89efc67dfdf5d011bf5f4250b985bd75287254766ced757c929b9c12a305b15\": container with ID starting with b89efc67dfdf5d011bf5f4250b985bd75287254766ced757c929b9c12a305b15 not found: ID does not exist" containerID="b89efc67dfdf5d011bf5f4250b985bd75287254766ced757c929b9c12a305b15"
Sep 17 17:48:40.734733 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:48:40.734686    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b89efc67dfdf5d011bf5f4250b985bd75287254766ced757c929b9c12a305b15"} err="failed to get container status \"b89efc67dfdf5d011bf5f4250b985bd75287254766ced757c929b9c12a305b15\": rpc error: code = NotFound desc = could not find container \"b89efc67dfdf5d011bf5f4250b985bd75287254766ced757c929b9c12a305b15\": container with ID starting with b89efc67dfdf5d011bf5f4250b985bd75287254766ced757c929b9c12a305b15 not found: ID does not exist"
Sep 17 17:48:40.734733 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:48:40.734701    2574 scope.go:117] "RemoveContainer" containerID="44ea00c134d5720335b050b1c812bd3613fc95cede151e9f556d2ca04fa76e9d"
Sep 17 17:48:40.734944 ip-10-0-134-96 kubenswrapper[2574]: E0917 17:48:40.734931    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"44ea00c134d5720335b050b1c812bd3613fc95cede151e9f556d2ca04fa76e9d\": container with ID starting with 44ea00c134d5720335b050b1c812bd3613fc95cede151e9f556d2ca04fa76e9d not found: ID does not exist" containerID="44ea00c134d5720335b050b1c812bd3613fc95cede151e9f556d2ca04fa76e9d"
Sep 17 17:48:40.735000 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:48:40.734949    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44ea00c134d5720335b050b1c812bd3613fc95cede151e9f556d2ca04fa76e9d"} err="failed to get container status \"44ea00c134d5720335b050b1c812bd3613fc95cede151e9f556d2ca04fa76e9d\": rpc error: code = NotFound desc = could not find container \"44ea00c134d5720335b050b1c812bd3613fc95cede151e9f556d2ca04fa76e9d\": container with ID starting with 44ea00c134d5720335b050b1c812bd3613fc95cede151e9f556d2ca04fa76e9d not found: ID does not exist"
Sep 17 17:48:40.735000 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:48:40.734961    2574 scope.go:117] "RemoveContainer" containerID="427d01bf4d7b1506317a01dbadf05887b4d829b978a3c3e5e7fc829bf4998a97"
Sep 17 17:48:40.735216 ip-10-0-134-96 kubenswrapper[2574]: E0917 17:48:40.735202    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"427d01bf4d7b1506317a01dbadf05887b4d829b978a3c3e5e7fc829bf4998a97\": container with ID starting with 427d01bf4d7b1506317a01dbadf05887b4d829b978a3c3e5e7fc829bf4998a97 not found: ID does not exist" containerID="427d01bf4d7b1506317a01dbadf05887b4d829b978a3c3e5e7fc829bf4998a97"
Sep 17 17:48:40.735263 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:48:40.735218    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"427d01bf4d7b1506317a01dbadf05887b4d829b978a3c3e5e7fc829bf4998a97"} err="failed to get container status \"427d01bf4d7b1506317a01dbadf05887b4d829b978a3c3e5e7fc829bf4998a97\": rpc error: code = NotFound desc = could not find container \"427d01bf4d7b1506317a01dbadf05887b4d829b978a3c3e5e7fc829bf4998a97\": container with ID starting with 427d01bf4d7b1506317a01dbadf05887b4d829b978a3c3e5e7fc829bf4998a97 not found: ID does not exist"
Sep 17 17:48:40.823019 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:48:40.822996    2574 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b5d1c8b1-6c52-4981-925d-0d53b4416cad-kserve-provision-location\") on node \"ip-10-0-134-96.ec2.internal\" DevicePath \"\""
Sep 17 17:48:40.823019 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:48:40.823017    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jvnvl\" (UniqueName: \"kubernetes.io/projected/b5d1c8b1-6c52-4981-925d-0d53b4416cad-kube-api-access-jvnvl\") on node \"ip-10-0-134-96.ec2.internal\" DevicePath \"\""
Sep 17 17:48:40.823184 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:48:40.823027    2574 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b5d1c8b1-6c52-4981-925d-0d53b4416cad-proxy-tls\") on node \"ip-10-0-134-96.ec2.internal\" DevicePath \"\""
Sep 17 17:48:41.062733 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:48:41.062703    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-51535-predictor-7bb954b865-jzlb4"]
Sep 17 17:48:41.066628 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:48:41.066606    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-51535-predictor-7bb954b865-jzlb4"]
Sep 17 17:48:41.195306 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:48:41.195282    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-51535-predictor-574dd6d475-p6srw"
Sep 17 17:48:41.225171 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:48:41.225103    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5f644bf6-20a1-4abd-b7cf-23757cbb6f75-kserve-provision-location\") pod \"5f644bf6-20a1-4abd-b7cf-23757cbb6f75\" (UID: \"5f644bf6-20a1-4abd-b7cf-23757cbb6f75\") "
Sep 17 17:48:41.225171 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:48:41.225166    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5f644bf6-20a1-4abd-b7cf-23757cbb6f75-kube-rbac-proxy-sar-config\") pod \"5f644bf6-20a1-4abd-b7cf-23757cbb6f75\" (UID: \"5f644bf6-20a1-4abd-b7cf-23757cbb6f75\") "
Sep 17 17:48:41.225346 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:48:41.225185    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5f644bf6-20a1-4abd-b7cf-23757cbb6f75-cabundle-cert\") pod \"5f644bf6-20a1-4abd-b7cf-23757cbb6f75\" (UID: \"5f644bf6-20a1-4abd-b7cf-23757cbb6f75\") "
Sep 17 17:48:41.225346 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:48:41.225204    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zs2wg\" (UniqueName: \"kubernetes.io/projected/5f644bf6-20a1-4abd-b7cf-23757cbb6f75-kube-api-access-zs2wg\") pod \"5f644bf6-20a1-4abd-b7cf-23757cbb6f75\" (UID: \"5f644bf6-20a1-4abd-b7cf-23757cbb6f75\") "
Sep 17 17:48:41.225346 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:48:41.225242    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5f644bf6-20a1-4abd-b7cf-23757cbb6f75-proxy-tls\") pod \"5f644bf6-20a1-4abd-b7cf-23757cbb6f75\" (UID: \"5f644bf6-20a1-4abd-b7cf-23757cbb6f75\") "
Sep 17 17:48:41.225499 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:48:41.225473    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f644bf6-20a1-4abd-b7cf-23757cbb6f75-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "5f644bf6-20a1-4abd-b7cf-23757cbb6f75" (UID: "5f644bf6-20a1-4abd-b7cf-23757cbb6f75"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:48:41.225618 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:48:41.225597    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f644bf6-20a1-4abd-b7cf-23757cbb6f75-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "5f644bf6-20a1-4abd-b7cf-23757cbb6f75" (UID: "5f644bf6-20a1-4abd-b7cf-23757cbb6f75"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:48:41.225618 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:48:41.225607    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f644bf6-20a1-4abd-b7cf-23757cbb6f75-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "5f644bf6-20a1-4abd-b7cf-23757cbb6f75" (UID: "5f644bf6-20a1-4abd-b7cf-23757cbb6f75"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:48:41.227221 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:48:41.227198    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5f644bf6-20a1-4abd-b7cf-23757cbb6f75-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "5f644bf6-20a1-4abd-b7cf-23757cbb6f75" (UID: "5f644bf6-20a1-4abd-b7cf-23757cbb6f75"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:48:41.227284 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:48:41.227247    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f644bf6-20a1-4abd-b7cf-23757cbb6f75-kube-api-access-zs2wg" (OuterVolumeSpecName: "kube-api-access-zs2wg") pod "5f644bf6-20a1-4abd-b7cf-23757cbb6f75" (UID: "5f644bf6-20a1-4abd-b7cf-23757cbb6f75"). InnerVolumeSpecName "kube-api-access-zs2wg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:48:41.326101 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:48:41.326067    2574 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5f644bf6-20a1-4abd-b7cf-23757cbb6f75-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-96.ec2.internal\" DevicePath \"\""
Sep 17 17:48:41.326101 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:48:41.326096    2574 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5f644bf6-20a1-4abd-b7cf-23757cbb6f75-cabundle-cert\") on node \"ip-10-0-134-96.ec2.internal\" DevicePath \"\""
Sep 17 17:48:41.326101 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:48:41.326105    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zs2wg\" (UniqueName: \"kubernetes.io/projected/5f644bf6-20a1-4abd-b7cf-23757cbb6f75-kube-api-access-zs2wg\") on node \"ip-10-0-134-96.ec2.internal\" DevicePath \"\""
Sep 17 17:48:41.326313 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:48:41.326114    2574 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5f644bf6-20a1-4abd-b7cf-23757cbb6f75-proxy-tls\") on node \"ip-10-0-134-96.ec2.internal\" DevicePath \"\""
Sep 17 17:48:41.326313 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:48:41.326123    2574 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5f644bf6-20a1-4abd-b7cf-23757cbb6f75-kserve-provision-location\") on node \"ip-10-0-134-96.ec2.internal\" DevicePath \"\""
Sep 17 17:48:41.704837 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:48:41.704803    2574 generic.go:356] "Generic (PLEG): container finished" podID="5f644bf6-20a1-4abd-b7cf-23757cbb6f75" containerID="de9069f2ba561ce5209b1f85f4dbca22bf523cef3fe256f677ed5e041144cbd5" exitCode=0
Sep 17 17:48:41.705323 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:48:41.704864    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-51535-predictor-574dd6d475-p6srw"
Sep 17 17:48:41.705323 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:48:41.704878    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-51535-predictor-574dd6d475-p6srw" event={"ID":"5f644bf6-20a1-4abd-b7cf-23757cbb6f75","Type":"ContainerDied","Data":"de9069f2ba561ce5209b1f85f4dbca22bf523cef3fe256f677ed5e041144cbd5"}
Sep 17 17:48:41.705323 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:48:41.704931    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-51535-predictor-574dd6d475-p6srw" event={"ID":"5f644bf6-20a1-4abd-b7cf-23757cbb6f75","Type":"ContainerDied","Data":"a1590d60151105c08cf34ecabae27f538070cad06079fe09efe2b676bb56848d"}
Sep 17 17:48:41.705323 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:48:41.704958    2574 scope.go:117] "RemoveContainer" containerID="c84a50fb176e9efb08c5bb2e4136f4fb0b157f15f0a5d9fc4695a991bf1d3e67"
Sep 17 17:48:41.713164 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:48:41.713143    2574 scope.go:117] "RemoveContainer" containerID="de9069f2ba561ce5209b1f85f4dbca22bf523cef3fe256f677ed5e041144cbd5"
Sep 17 17:48:41.719958 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:48:41.719942    2574 scope.go:117] "RemoveContainer" containerID="748a3655bda4902cefae13bf8949bd888fc6213bec66cd77ffc9f379f6a7eb9f"
Sep 17 17:48:41.727067 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:48:41.727030    2574 scope.go:117] "RemoveContainer" containerID="c84a50fb176e9efb08c5bb2e4136f4fb0b157f15f0a5d9fc4695a991bf1d3e67"
Sep 17 17:48:41.727317 ip-10-0-134-96 kubenswrapper[2574]: E0917 17:48:41.727301    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c84a50fb176e9efb08c5bb2e4136f4fb0b157f15f0a5d9fc4695a991bf1d3e67\": container with ID starting with c84a50fb176e9efb08c5bb2e4136f4fb0b157f15f0a5d9fc4695a991bf1d3e67 not found: ID does not exist" containerID="c84a50fb176e9efb08c5bb2e4136f4fb0b157f15f0a5d9fc4695a991bf1d3e67"
Sep 17 17:48:41.727363 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:48:41.727325    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c84a50fb176e9efb08c5bb2e4136f4fb0b157f15f0a5d9fc4695a991bf1d3e67"} err="failed to get container status \"c84a50fb176e9efb08c5bb2e4136f4fb0b157f15f0a5d9fc4695a991bf1d3e67\": rpc error: code = NotFound desc = could not find container \"c84a50fb176e9efb08c5bb2e4136f4fb0b157f15f0a5d9fc4695a991bf1d3e67\": container with ID starting with c84a50fb176e9efb08c5bb2e4136f4fb0b157f15f0a5d9fc4695a991bf1d3e67 not found: ID does not exist"
Sep 17 17:48:41.727363 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:48:41.727341    2574 scope.go:117] "RemoveContainer" containerID="de9069f2ba561ce5209b1f85f4dbca22bf523cef3fe256f677ed5e041144cbd5"
Sep 17 17:48:41.727532 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:48:41.727515    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-51535-predictor-574dd6d475-p6srw"]
Sep 17 17:48:41.727571 ip-10-0-134-96 kubenswrapper[2574]: E0917 17:48:41.727559    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de9069f2ba561ce5209b1f85f4dbca22bf523cef3fe256f677ed5e041144cbd5\": container with ID starting with de9069f2ba561ce5209b1f85f4dbca22bf523cef3fe256f677ed5e041144cbd5 not found: ID does not exist" containerID="de9069f2ba561ce5209b1f85f4dbca22bf523cef3fe256f677ed5e041144cbd5"
Sep 17 17:48:41.727608 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:48:41.727579    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de9069f2ba561ce5209b1f85f4dbca22bf523cef3fe256f677ed5e041144cbd5"} err="failed to get container status \"de9069f2ba561ce5209b1f85f4dbca22bf523cef3fe256f677ed5e041144cbd5\": rpc error: code = NotFound desc = could not find container \"de9069f2ba561ce5209b1f85f4dbca22bf523cef3fe256f677ed5e041144cbd5\": container with ID starting with de9069f2ba561ce5209b1f85f4dbca22bf523cef3fe256f677ed5e041144cbd5 not found: ID does not exist"
Sep 17 17:48:41.727608 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:48:41.727593    2574 scope.go:117] "RemoveContainer" containerID="748a3655bda4902cefae13bf8949bd888fc6213bec66cd77ffc9f379f6a7eb9f"
Sep 17 17:48:41.727824 ip-10-0-134-96 kubenswrapper[2574]: E0917 17:48:41.727805    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"748a3655bda4902cefae13bf8949bd888fc6213bec66cd77ffc9f379f6a7eb9f\": container with ID starting with 748a3655bda4902cefae13bf8949bd888fc6213bec66cd77ffc9f379f6a7eb9f not found: ID does not exist" containerID="748a3655bda4902cefae13bf8949bd888fc6213bec66cd77ffc9f379f6a7eb9f"
Sep 17 17:48:41.727881 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:48:41.727832    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"748a3655bda4902cefae13bf8949bd888fc6213bec66cd77ffc9f379f6a7eb9f"} err="failed to get container status \"748a3655bda4902cefae13bf8949bd888fc6213bec66cd77ffc9f379f6a7eb9f\": rpc error: code = NotFound desc = could not find container \"748a3655bda4902cefae13bf8949bd888fc6213bec66cd77ffc9f379f6a7eb9f\": container with ID starting with 748a3655bda4902cefae13bf8949bd888fc6213bec66cd77ffc9f379f6a7eb9f not found: ID does not exist"
Sep 17 17:48:41.730796 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:48:41.730776    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-51535-predictor-574dd6d475-p6srw"]
Sep 17 17:48:42.953435 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:48:42.953406    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5f644bf6-20a1-4abd-b7cf-23757cbb6f75" path="/var/lib/kubelet/pods/5f644bf6-20a1-4abd-b7cf-23757cbb6f75/volumes"
Sep 17 17:48:42.953867 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:48:42.953854    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b5d1c8b1-6c52-4981-925d-0d53b4416cad" path="/var/lib/kubelet/pods/b5d1c8b1-6c52-4981-925d-0d53b4416cad/volumes"
Sep 17 17:48:56.608554 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:48:56.608484    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j4ds_d3f553e8-b416-4d7d-bbef-991a0404b287/konnectivity-agent/0.log" limit=1024
Sep 17 17:48:56.772197 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:48:56.772169    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j4ds_d3f553e8-b416-4d7d-bbef-991a0404b287/konnectivity-agent/0.log" limit=1024
Sep 17 17:49:06.859679 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:49:06.859651    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-45dp8_bd527736-9920-4c79-9069-1eb8be07f869/ovn-acl-logging/0.log"
Sep 17 17:49:06.860709 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:49:06.860685    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-45dp8_bd527736-9920-4c79-9069-1eb8be07f869/ovn-acl-logging/0.log"
Sep 17 17:49:51.429409 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:49:51.429378    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j4ds_d3f553e8-b416-4d7d-bbef-991a0404b287/konnectivity-agent/0.log" limit=1024
Sep 17 17:49:51.450922 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:49:51.450892    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-46f911-predictor-78d489899c-8bg7h"]
Sep 17 17:49:51.451243 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:49:51.451226    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b5d1c8b1-6c52-4981-925d-0d53b4416cad" containerName="kube-rbac-proxy"
Sep 17 17:49:51.451324 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:49:51.451245    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="b5d1c8b1-6c52-4981-925d-0d53b4416cad" containerName="kube-rbac-proxy"
Sep 17 17:49:51.451324 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:49:51.451260    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9e640c3f-798b-4734-9258-361fb5ccde21" containerName="storage-initializer"
Sep 17 17:49:51.451324 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:49:51.451269    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="9e640c3f-798b-4734-9258-361fb5ccde21" containerName="storage-initializer"
Sep 17 17:49:51.451324 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:49:51.451287    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d4b9e97c-cfed-4de6-b410-4d8b1f3c98b5" containerName="kube-rbac-proxy"
Sep 17 17:49:51.451324 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:49:51.451294    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="d4b9e97c-cfed-4de6-b410-4d8b1f3c98b5" containerName="kube-rbac-proxy"
Sep 17 17:49:51.451324 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:49:51.451306    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9e640c3f-798b-4734-9258-361fb5ccde21" containerName="kserve-container"
Sep 17 17:49:51.451324 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:49:51.451314    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="9e640c3f-798b-4734-9258-361fb5ccde21" containerName="kserve-container"
Sep 17 17:49:51.451324 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:49:51.451322    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5f644bf6-20a1-4abd-b7cf-23757cbb6f75" containerName="kserve-container"
Sep 17 17:49:51.451708 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:49:51.451331    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="5f644bf6-20a1-4abd-b7cf-23757cbb6f75" containerName="kserve-container"
Sep 17 17:49:51.451708 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:49:51.451342    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b5d1c8b1-6c52-4981-925d-0d53b4416cad" containerName="storage-initializer"
Sep 17 17:49:51.451708 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:49:51.451350    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="b5d1c8b1-6c52-4981-925d-0d53b4416cad" containerName="storage-initializer"
Sep 17 17:49:51.451708 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:49:51.451359    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d4b9e97c-cfed-4de6-b410-4d8b1f3c98b5" containerName="storage-initializer"
Sep 17 17:49:51.451708 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:49:51.451368    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="d4b9e97c-cfed-4de6-b410-4d8b1f3c98b5" containerName="storage-initializer"
Sep 17 17:49:51.451708 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:49:51.451385    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5f644bf6-20a1-4abd-b7cf-23757cbb6f75" containerName="storage-initializer"
Sep 17 17:49:51.451708 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:49:51.451393    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="5f644bf6-20a1-4abd-b7cf-23757cbb6f75" containerName="storage-initializer"
Sep 17 17:49:51.451708 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:49:51.451401    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5f644bf6-20a1-4abd-b7cf-23757cbb6f75" containerName="kube-rbac-proxy"
Sep 17 17:49:51.451708 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:49:51.451409    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="5f644bf6-20a1-4abd-b7cf-23757cbb6f75" containerName="kube-rbac-proxy"
Sep 17 17:49:51.451708 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:49:51.451417    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9e640c3f-798b-4734-9258-361fb5ccde21" containerName="kube-rbac-proxy"
Sep 17 17:49:51.451708 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:49:51.451425    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="9e640c3f-798b-4734-9258-361fb5ccde21" containerName="kube-rbac-proxy"
Sep 17 17:49:51.451708 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:49:51.451438    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b5d1c8b1-6c52-4981-925d-0d53b4416cad" containerName="kserve-container"
Sep 17 17:49:51.451708 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:49:51.451448    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="b5d1c8b1-6c52-4981-925d-0d53b4416cad" containerName="kserve-container"
Sep 17 17:49:51.451708 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:49:51.451460    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d4b9e97c-cfed-4de6-b410-4d8b1f3c98b5" containerName="kserve-container"
Sep 17 17:49:51.451708 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:49:51.451468    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="d4b9e97c-cfed-4de6-b410-4d8b1f3c98b5" containerName="kserve-container"
Sep 17 17:49:51.451708 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:49:51.451539    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="d4b9e97c-cfed-4de6-b410-4d8b1f3c98b5" containerName="kserve-container"
Sep 17 17:49:51.451708 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:49:51.451549    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="9e640c3f-798b-4734-9258-361fb5ccde21" containerName="kserve-container"
Sep 17 17:49:51.451708 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:49:51.451560    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="5f644bf6-20a1-4abd-b7cf-23757cbb6f75" containerName="kube-rbac-proxy"
Sep 17 17:49:51.451708 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:49:51.451570    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="9e640c3f-798b-4734-9258-361fb5ccde21" containerName="kube-rbac-proxy"
Sep 17 17:49:51.451708 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:49:51.451579    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="5f644bf6-20a1-4abd-b7cf-23757cbb6f75" containerName="kserve-container"
Sep 17 17:49:51.451708 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:49:51.451589    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="b5d1c8b1-6c52-4981-925d-0d53b4416cad" containerName="kube-rbac-proxy"
Sep 17 17:49:51.451708 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:49:51.451602    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="d4b9e97c-cfed-4de6-b410-4d8b1f3c98b5" containerName="kube-rbac-proxy"
Sep 17 17:49:51.451708 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:49:51.451611    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="b5d1c8b1-6c52-4981-925d-0d53b4416cad" containerName="kserve-container"
Sep 17 17:49:51.454822 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:49:51.454804    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-46f911-predictor-78d489899c-8bg7h"
Sep 17 17:49:51.457300 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:49:51.457276    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-secondary-46f911-predictor-serving-cert\""
Sep 17 17:49:51.457300 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:49:51.457277    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 17:49:51.457460 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:49:51.457345    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"invalid-s3-secret-46f911\""
Sep 17 17:49:51.457460 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:49:51.457408    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"invalid-s3-sa-46f911-dockercfg-qdrdf\""
Sep 17 17:49:51.457460 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:49:51.457420    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 17:49:51.462750 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:49:51.462730    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 17:49:51.462863 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:49:51.462767    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-secondary-46f911-kube-rbac-proxy-sar-config\""
Sep 17 17:49:51.469177 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:49:51.469159    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-46f911-predictor-78d489899c-8bg7h"]
Sep 17 17:49:51.647359 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:49:51.647332    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8e517296-908e-4b12-931a-cf59260426dc-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-46f911-predictor-78d489899c-8bg7h\" (UID: \"8e517296-908e-4b12-931a-cf59260426dc\") " pod="kserve-ci-e2e-test/isvc-secondary-46f911-predictor-78d489899c-8bg7h"
Sep 17 17:49:51.647518 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:49:51.647378    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8e517296-908e-4b12-931a-cf59260426dc-proxy-tls\") pod \"isvc-secondary-46f911-predictor-78d489899c-8bg7h\" (UID: \"8e517296-908e-4b12-931a-cf59260426dc\") " pod="kserve-ci-e2e-test/isvc-secondary-46f911-predictor-78d489899c-8bg7h"
Sep 17 17:49:51.647518 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:49:51.647402    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ng688\" (UniqueName: \"kubernetes.io/projected/8e517296-908e-4b12-931a-cf59260426dc-kube-api-access-ng688\") pod \"isvc-secondary-46f911-predictor-78d489899c-8bg7h\" (UID: \"8e517296-908e-4b12-931a-cf59260426dc\") " pod="kserve-ci-e2e-test/isvc-secondary-46f911-predictor-78d489899c-8bg7h"
Sep 17 17:49:51.647518 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:49:51.647446    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8e517296-908e-4b12-931a-cf59260426dc-cabundle-cert\") pod \"isvc-secondary-46f911-predictor-78d489899c-8bg7h\" (UID: \"8e517296-908e-4b12-931a-cf59260426dc\") " pod="kserve-ci-e2e-test/isvc-secondary-46f911-predictor-78d489899c-8bg7h"
Sep 17 17:49:51.647643 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:49:51.647540    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8e517296-908e-4b12-931a-cf59260426dc-kserve-provision-location\") pod \"isvc-secondary-46f911-predictor-78d489899c-8bg7h\" (UID: \"8e517296-908e-4b12-931a-cf59260426dc\") " pod="kserve-ci-e2e-test/isvc-secondary-46f911-predictor-78d489899c-8bg7h"
Sep 17 17:49:51.720492 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:49:51.720424    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j4ds_d3f553e8-b416-4d7d-bbef-991a0404b287/konnectivity-agent/0.log" limit=1024
Sep 17 17:49:51.748812 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:49:51.748786    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8e517296-908e-4b12-931a-cf59260426dc-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-46f911-predictor-78d489899c-8bg7h\" (UID: \"8e517296-908e-4b12-931a-cf59260426dc\") " pod="kserve-ci-e2e-test/isvc-secondary-46f911-predictor-78d489899c-8bg7h"
Sep 17 17:49:51.748919 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:49:51.748821    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8e517296-908e-4b12-931a-cf59260426dc-proxy-tls\") pod \"isvc-secondary-46f911-predictor-78d489899c-8bg7h\" (UID: \"8e517296-908e-4b12-931a-cf59260426dc\") " pod="kserve-ci-e2e-test/isvc-secondary-46f911-predictor-78d489899c-8bg7h"
Sep 17 17:49:51.748919 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:49:51.748838    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ng688\" (UniqueName: \"kubernetes.io/projected/8e517296-908e-4b12-931a-cf59260426dc-kube-api-access-ng688\") pod \"isvc-secondary-46f911-predictor-78d489899c-8bg7h\" (UID: \"8e517296-908e-4b12-931a-cf59260426dc\") " pod="kserve-ci-e2e-test/isvc-secondary-46f911-predictor-78d489899c-8bg7h"
Sep 17 17:49:51.748919 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:49:51.748866    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8e517296-908e-4b12-931a-cf59260426dc-cabundle-cert\") pod \"isvc-secondary-46f911-predictor-78d489899c-8bg7h\" (UID: \"8e517296-908e-4b12-931a-cf59260426dc\") " pod="kserve-ci-e2e-test/isvc-secondary-46f911-predictor-78d489899c-8bg7h"
Sep 17 17:49:51.749095 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:49:51.748922    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8e517296-908e-4b12-931a-cf59260426dc-kserve-provision-location\") pod \"isvc-secondary-46f911-predictor-78d489899c-8bg7h\" (UID: \"8e517296-908e-4b12-931a-cf59260426dc\") " pod="kserve-ci-e2e-test/isvc-secondary-46f911-predictor-78d489899c-8bg7h"
Sep 17 17:49:51.749448 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:49:51.749422    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8e517296-908e-4b12-931a-cf59260426dc-kserve-provision-location\") pod \"isvc-secondary-46f911-predictor-78d489899c-8bg7h\" (UID: \"8e517296-908e-4b12-931a-cf59260426dc\") " pod="kserve-ci-e2e-test/isvc-secondary-46f911-predictor-78d489899c-8bg7h"
Sep 17 17:49:51.749448 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:49:51.749443    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8e517296-908e-4b12-931a-cf59260426dc-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-46f911-predictor-78d489899c-8bg7h\" (UID: \"8e517296-908e-4b12-931a-cf59260426dc\") " pod="kserve-ci-e2e-test/isvc-secondary-46f911-predictor-78d489899c-8bg7h"
Sep 17 17:49:51.749576 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:49:51.749561    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8e517296-908e-4b12-931a-cf59260426dc-cabundle-cert\") pod \"isvc-secondary-46f911-predictor-78d489899c-8bg7h\" (UID: \"8e517296-908e-4b12-931a-cf59260426dc\") " pod="kserve-ci-e2e-test/isvc-secondary-46f911-predictor-78d489899c-8bg7h"
Sep 17 17:49:51.751429 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:49:51.751401    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8e517296-908e-4b12-931a-cf59260426dc-proxy-tls\") pod \"isvc-secondary-46f911-predictor-78d489899c-8bg7h\" (UID: \"8e517296-908e-4b12-931a-cf59260426dc\") " pod="kserve-ci-e2e-test/isvc-secondary-46f911-predictor-78d489899c-8bg7h"
Sep 17 17:49:51.757570 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:49:51.757548    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ng688\" (UniqueName: \"kubernetes.io/projected/8e517296-908e-4b12-931a-cf59260426dc-kube-api-access-ng688\") pod \"isvc-secondary-46f911-predictor-78d489899c-8bg7h\" (UID: \"8e517296-908e-4b12-931a-cf59260426dc\") " pod="kserve-ci-e2e-test/isvc-secondary-46f911-predictor-78d489899c-8bg7h"
Sep 17 17:49:51.769298 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:49:51.769282    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-46f911-predictor-78d489899c-8bg7h"
Sep 17 17:49:52.039456 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:49:52.039428    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-46f911-predictor-78d489899c-8bg7h"]
Sep 17 17:49:52.042193 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:49:52.042169    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e517296_908e_4b12_931a_cf59260426dc.slice/crio-5c72078ac96c9e0ec4c75d86c9c3c083d227005db2fdb757ff2912a5fd51e3aa WatchSource:0}: Error finding container 5c72078ac96c9e0ec4c75d86c9c3c083d227005db2fdb757ff2912a5fd51e3aa: Status 404 returned error can't find the container with id 5c72078ac96c9e0ec4c75d86c9c3c083d227005db2fdb757ff2912a5fd51e3aa
Sep 17 17:49:52.977221 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:49:52.977187    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-46f911-predictor-78d489899c-8bg7h" event={"ID":"8e517296-908e-4b12-931a-cf59260426dc","Type":"ContainerStarted","Data":"17cad68e9813bddf5edfa0d914b28ed03cf4d7bc503d6f02b7f41653ab809a82"}
Sep 17 17:49:52.977597 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:49:52.977240    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-46f911-predictor-78d489899c-8bg7h" event={"ID":"8e517296-908e-4b12-931a-cf59260426dc","Type":"ContainerStarted","Data":"5c72078ac96c9e0ec4c75d86c9c3c083d227005db2fdb757ff2912a5fd51e3aa"}
Sep 17 17:49:53.982514 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:49:53.982491    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-46f911-predictor-78d489899c-8bg7h_8e517296-908e-4b12-931a-cf59260426dc/storage-initializer/0.log"
Sep 17 17:49:53.982803 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:49:53.982526    2574 generic.go:356] "Generic (PLEG): container finished" podID="8e517296-908e-4b12-931a-cf59260426dc" containerID="17cad68e9813bddf5edfa0d914b28ed03cf4d7bc503d6f02b7f41653ab809a82" exitCode=1
Sep 17 17:49:53.982803 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:49:53.982599    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-46f911-predictor-78d489899c-8bg7h" event={"ID":"8e517296-908e-4b12-931a-cf59260426dc","Type":"ContainerDied","Data":"17cad68e9813bddf5edfa0d914b28ed03cf4d7bc503d6f02b7f41653ab809a82"}
Sep 17 17:49:54.987795 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:49:54.987770    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-46f911-predictor-78d489899c-8bg7h_8e517296-908e-4b12-931a-cf59260426dc/storage-initializer/0.log"
Sep 17 17:49:54.988192 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:49:54.987849    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-46f911-predictor-78d489899c-8bg7h" event={"ID":"8e517296-908e-4b12-931a-cf59260426dc","Type":"ContainerStarted","Data":"0b370948c7f521cbe7d380523491d484895b7a27a60eb64e52ca545be7c7791d"}
Sep 17 17:49:56.997402 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:49:56.997379    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-46f911-predictor-78d489899c-8bg7h_8e517296-908e-4b12-931a-cf59260426dc/storage-initializer/1.log"
Sep 17 17:49:56.997807 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:49:56.997792    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-46f911-predictor-78d489899c-8bg7h_8e517296-908e-4b12-931a-cf59260426dc/storage-initializer/0.log"
Sep 17 17:49:56.997854 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:49:56.997823    2574 generic.go:356] "Generic (PLEG): container finished" podID="8e517296-908e-4b12-931a-cf59260426dc" containerID="0b370948c7f521cbe7d380523491d484895b7a27a60eb64e52ca545be7c7791d" exitCode=1
Sep 17 17:49:56.997894 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:49:56.997874    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-46f911-predictor-78d489899c-8bg7h" event={"ID":"8e517296-908e-4b12-931a-cf59260426dc","Type":"ContainerDied","Data":"0b370948c7f521cbe7d380523491d484895b7a27a60eb64e52ca545be7c7791d"}
Sep 17 17:49:56.997945 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:49:56.997920    2574 scope.go:117] "RemoveContainer" containerID="17cad68e9813bddf5edfa0d914b28ed03cf4d7bc503d6f02b7f41653ab809a82"
Sep 17 17:49:56.998210 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:49:56.998186    2574 scope.go:117] "RemoveContainer" containerID="17cad68e9813bddf5edfa0d914b28ed03cf4d7bc503d6f02b7f41653ab809a82"
Sep 17 17:49:57.008096 ip-10-0-134-96 kubenswrapper[2574]: E0917 17:49:57.008068    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"storage-initializer\" with CrashLoopBackOff: \"back-off 10s restarting failed container=storage-initializer pod=isvc-secondary-46f911-predictor-78d489899c-8bg7h_kserve-ci-e2e-test(8e517296-908e-4b12-931a-cf59260426dc)\"" pod="kserve-ci-e2e-test/isvc-secondary-46f911-predictor-78d489899c-8bg7h" podUID="8e517296-908e-4b12-931a-cf59260426dc"
Sep 17 17:49:57.008387 ip-10-0-134-96 kubenswrapper[2574]: E0917 17:49:57.008364    2574 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-secondary-46f911-predictor-78d489899c-8bg7h_kserve-ci-e2e-test_8e517296-908e-4b12-931a-cf59260426dc_0 in pod sandbox 5c72078ac96c9e0ec4c75d86c9c3c083d227005db2fdb757ff2912a5fd51e3aa from index: no such id: '17cad68e9813bddf5edfa0d914b28ed03cf4d7bc503d6f02b7f41653ab809a82'" containerID="17cad68e9813bddf5edfa0d914b28ed03cf4d7bc503d6f02b7f41653ab809a82"
Sep 17 17:49:57.008436 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:49:57.008397    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"17cad68e9813bddf5edfa0d914b28ed03cf4d7bc503d6f02b7f41653ab809a82"} err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-secondary-46f911-predictor-78d489899c-8bg7h_kserve-ci-e2e-test_8e517296-908e-4b12-931a-cf59260426dc_0 in pod sandbox 5c72078ac96c9e0ec4c75d86c9c3c083d227005db2fdb757ff2912a5fd51e3aa from index: no such id: '17cad68e9813bddf5edfa0d914b28ed03cf4d7bc503d6f02b7f41653ab809a82'"
Sep 17 17:49:58.003604 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:49:58.003582    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-46f911-predictor-78d489899c-8bg7h_8e517296-908e-4b12-931a-cf59260426dc/storage-initializer/1.log"
Sep 17 17:50:05.457286 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:50:05.457255    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-46f911-predictor-78d489899c-8bg7h"]
Sep 17 17:50:05.518793 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:50:05.518764    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j4ds_d3f553e8-b416-4d7d-bbef-991a0404b287/konnectivity-agent/0.log" limit=1024
Sep 17 17:50:05.577360 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:50:05.577342    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-46f911-predictor-78d489899c-8bg7h_8e517296-908e-4b12-931a-cf59260426dc/storage-initializer/1.log"
Sep 17 17:50:05.582524 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:50:05.582507    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-46f911-predictor-78d489899c-8bg7h"
Sep 17 17:50:05.645066 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:50:05.645022    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8e517296-908e-4b12-931a-cf59260426dc-cabundle-cert\") pod \"8e517296-908e-4b12-931a-cf59260426dc\" (UID: \"8e517296-908e-4b12-931a-cf59260426dc\") "
Sep 17 17:50:05.645175 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:50:05.645084    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8e517296-908e-4b12-931a-cf59260426dc-kserve-provision-location\") pod \"8e517296-908e-4b12-931a-cf59260426dc\" (UID: \"8e517296-908e-4b12-931a-cf59260426dc\") "
Sep 17 17:50:05.645175 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:50:05.645118    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8e517296-908e-4b12-931a-cf59260426dc-kube-rbac-proxy-sar-config\") pod \"8e517296-908e-4b12-931a-cf59260426dc\" (UID: \"8e517296-908e-4b12-931a-cf59260426dc\") "
Sep 17 17:50:05.645175 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:50:05.645160    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ng688\" (UniqueName: \"kubernetes.io/projected/8e517296-908e-4b12-931a-cf59260426dc-kube-api-access-ng688\") pod \"8e517296-908e-4b12-931a-cf59260426dc\" (UID: \"8e517296-908e-4b12-931a-cf59260426dc\") "
Sep 17 17:50:05.645335 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:50:05.645226    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8e517296-908e-4b12-931a-cf59260426dc-proxy-tls\") pod \"8e517296-908e-4b12-931a-cf59260426dc\" (UID: \"8e517296-908e-4b12-931a-cf59260426dc\") "
Sep 17 17:50:05.645379 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:50:05.645334    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e517296-908e-4b12-931a-cf59260426dc-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "8e517296-908e-4b12-931a-cf59260426dc" (UID: "8e517296-908e-4b12-931a-cf59260426dc"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:50:05.645436 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:50:05.645393    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e517296-908e-4b12-931a-cf59260426dc-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "8e517296-908e-4b12-931a-cf59260426dc" (UID: "8e517296-908e-4b12-931a-cf59260426dc"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:50:05.645436 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:50:05.645421    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e517296-908e-4b12-931a-cf59260426dc-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "8e517296-908e-4b12-931a-cf59260426dc" (UID: "8e517296-908e-4b12-931a-cf59260426dc"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:50:05.645525 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:50:05.645465    2574 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8e517296-908e-4b12-931a-cf59260426dc-cabundle-cert\") on node \"ip-10-0-134-96.ec2.internal\" DevicePath \"\""
Sep 17 17:50:05.645525 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:50:05.645482    2574 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8e517296-908e-4b12-931a-cf59260426dc-kserve-provision-location\") on node \"ip-10-0-134-96.ec2.internal\" DevicePath \"\""
Sep 17 17:50:05.647178 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:50:05.647155    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e517296-908e-4b12-931a-cf59260426dc-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "8e517296-908e-4b12-931a-cf59260426dc" (UID: "8e517296-908e-4b12-931a-cf59260426dc"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:50:05.647273 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:50:05.647181    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e517296-908e-4b12-931a-cf59260426dc-kube-api-access-ng688" (OuterVolumeSpecName: "kube-api-access-ng688") pod "8e517296-908e-4b12-931a-cf59260426dc" (UID: "8e517296-908e-4b12-931a-cf59260426dc"). InnerVolumeSpecName "kube-api-access-ng688". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:50:05.746776 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:50:05.746755    2574 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8e517296-908e-4b12-931a-cf59260426dc-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-96.ec2.internal\" DevicePath \"\""
Sep 17 17:50:05.746776 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:50:05.746776    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ng688\" (UniqueName: \"kubernetes.io/projected/8e517296-908e-4b12-931a-cf59260426dc-kube-api-access-ng688\") on node \"ip-10-0-134-96.ec2.internal\" DevicePath \"\""
Sep 17 17:50:05.746906 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:50:05.746785    2574 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8e517296-908e-4b12-931a-cf59260426dc-proxy-tls\") on node \"ip-10-0-134-96.ec2.internal\" DevicePath \"\""
Sep 17 17:50:06.034119 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:50:06.034037    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-46f911-predictor-78d489899c-8bg7h_8e517296-908e-4b12-931a-cf59260426dc/storage-initializer/1.log"
Sep 17 17:50:06.034273 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:50:06.034154    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-46f911-predictor-78d489899c-8bg7h"
Sep 17 17:50:06.034273 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:50:06.034196    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-46f911-predictor-78d489899c-8bg7h" event={"ID":"8e517296-908e-4b12-931a-cf59260426dc","Type":"ContainerDied","Data":"5c72078ac96c9e0ec4c75d86c9c3c083d227005db2fdb757ff2912a5fd51e3aa"}
Sep 17 17:50:06.034273 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:50:06.034268    2574 scope.go:117] "RemoveContainer" containerID="0b370948c7f521cbe7d380523491d484895b7a27a60eb64e52ca545be7c7791d"
Sep 17 17:50:06.083203 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:50:06.083178    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-46f911-predictor-78d489899c-8bg7h"]
Sep 17 17:50:06.088892 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:50:06.088870    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-46f911-predictor-78d489899c-8bg7h"]
Sep 17 17:50:06.952876 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:50:06.952847    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8e517296-908e-4b12-931a-cf59260426dc" path="/var/lib/kubelet/pods/8e517296-908e-4b12-931a-cf59260426dc/volumes"
Sep 17 17:50:07.002090 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:50:07.002063    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j4ds_d3f553e8-b416-4d7d-bbef-991a0404b287/konnectivity-agent/0.log" limit=1024
Sep 17 17:50:10.757208 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:50:10.757177    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j4ds_d3f553e8-b416-4d7d-bbef-991a0404b287/konnectivity-agent/0.log" limit=1024
Sep 17 17:50:10.758249 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:50:10.758225    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-385bf1-predictor-644d7ccfd9-47sc9"]
Sep 17 17:50:10.758578 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:50:10.758556    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8e517296-908e-4b12-931a-cf59260426dc" containerName="storage-initializer"
Sep 17 17:50:10.758650 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:50:10.758583    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="8e517296-908e-4b12-931a-cf59260426dc" containerName="storage-initializer"
Sep 17 17:50:10.758650 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:50:10.758599    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8e517296-908e-4b12-931a-cf59260426dc" containerName="storage-initializer"
Sep 17 17:50:10.758650 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:50:10.758609    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="8e517296-908e-4b12-931a-cf59260426dc" containerName="storage-initializer"
Sep 17 17:50:10.758805 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:50:10.758699    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="8e517296-908e-4b12-931a-cf59260426dc" containerName="storage-initializer"
Sep 17 17:50:10.758866 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:50:10.758844    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="8e517296-908e-4b12-931a-cf59260426dc" containerName="storage-initializer"
Sep 17 17:50:10.763163 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:50:10.763143    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-385bf1-predictor-644d7ccfd9-47sc9"
Sep 17 17:50:10.765832 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:50:10.765811    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"fail-s3-secret-385bf1\""
Sep 17 17:50:10.765933 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:50:10.765849    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 17:50:10.765933 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:50:10.765884    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 17:50:10.766068 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:50:10.765971    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 17:50:10.766147 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:50:10.766125    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"fail-s3-sa-385bf1-dockercfg-9qj9g\""
Sep 17 17:50:10.766286 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:50:10.766193    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-init-fail-385bf1-predictor-serving-cert\""
Sep 17 17:50:10.766814 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:50:10.766795    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-init-fail-385bf1-kube-rbac-proxy-sar-config\""
Sep 17 17:50:10.771998 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:50:10.771980    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-385bf1-predictor-644d7ccfd9-47sc9"]
Sep 17 17:50:10.779968 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:50:10.779947    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/64ac92d0-3d22-415c-abab-8405fb594950-cabundle-cert\") pod \"isvc-init-fail-385bf1-predictor-644d7ccfd9-47sc9\" (UID: \"64ac92d0-3d22-415c-abab-8405fb594950\") " pod="kserve-ci-e2e-test/isvc-init-fail-385bf1-predictor-644d7ccfd9-47sc9"
Sep 17 17:50:10.780080 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:50:10.779978    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/64ac92d0-3d22-415c-abab-8405fb594950-kserve-provision-location\") pod \"isvc-init-fail-385bf1-predictor-644d7ccfd9-47sc9\" (UID: \"64ac92d0-3d22-415c-abab-8405fb594950\") " pod="kserve-ci-e2e-test/isvc-init-fail-385bf1-predictor-644d7ccfd9-47sc9"
Sep 17 17:50:10.780080 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:50:10.780008    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/64ac92d0-3d22-415c-abab-8405fb594950-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-385bf1-predictor-644d7ccfd9-47sc9\" (UID: \"64ac92d0-3d22-415c-abab-8405fb594950\") " pod="kserve-ci-e2e-test/isvc-init-fail-385bf1-predictor-644d7ccfd9-47sc9"
Sep 17 17:50:10.780204 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:50:10.780136    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/64ac92d0-3d22-415c-abab-8405fb594950-proxy-tls\") pod \"isvc-init-fail-385bf1-predictor-644d7ccfd9-47sc9\" (UID: \"64ac92d0-3d22-415c-abab-8405fb594950\") " pod="kserve-ci-e2e-test/isvc-init-fail-385bf1-predictor-644d7ccfd9-47sc9"
Sep 17 17:50:10.780204 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:50:10.780173    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8nq8f\" (UniqueName: \"kubernetes.io/projected/64ac92d0-3d22-415c-abab-8405fb594950-kube-api-access-8nq8f\") pod \"isvc-init-fail-385bf1-predictor-644d7ccfd9-47sc9\" (UID: \"64ac92d0-3d22-415c-abab-8405fb594950\") " pod="kserve-ci-e2e-test/isvc-init-fail-385bf1-predictor-644d7ccfd9-47sc9"
Sep 17 17:50:10.881029 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:50:10.881004    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/64ac92d0-3d22-415c-abab-8405fb594950-cabundle-cert\") pod \"isvc-init-fail-385bf1-predictor-644d7ccfd9-47sc9\" (UID: \"64ac92d0-3d22-415c-abab-8405fb594950\") " pod="kserve-ci-e2e-test/isvc-init-fail-385bf1-predictor-644d7ccfd9-47sc9"
Sep 17 17:50:10.881152 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:50:10.881038    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/64ac92d0-3d22-415c-abab-8405fb594950-kserve-provision-location\") pod \"isvc-init-fail-385bf1-predictor-644d7ccfd9-47sc9\" (UID: \"64ac92d0-3d22-415c-abab-8405fb594950\") " pod="kserve-ci-e2e-test/isvc-init-fail-385bf1-predictor-644d7ccfd9-47sc9"
Sep 17 17:50:10.881152 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:50:10.881099    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/64ac92d0-3d22-415c-abab-8405fb594950-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-385bf1-predictor-644d7ccfd9-47sc9\" (UID: \"64ac92d0-3d22-415c-abab-8405fb594950\") " pod="kserve-ci-e2e-test/isvc-init-fail-385bf1-predictor-644d7ccfd9-47sc9"
Sep 17 17:50:10.881152 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:50:10.881145    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/64ac92d0-3d22-415c-abab-8405fb594950-proxy-tls\") pod \"isvc-init-fail-385bf1-predictor-644d7ccfd9-47sc9\" (UID: \"64ac92d0-3d22-415c-abab-8405fb594950\") " pod="kserve-ci-e2e-test/isvc-init-fail-385bf1-predictor-644d7ccfd9-47sc9"
Sep 17 17:50:10.881287 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:50:10.881175    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8nq8f\" (UniqueName: \"kubernetes.io/projected/64ac92d0-3d22-415c-abab-8405fb594950-kube-api-access-8nq8f\") pod \"isvc-init-fail-385bf1-predictor-644d7ccfd9-47sc9\" (UID: \"64ac92d0-3d22-415c-abab-8405fb594950\") " pod="kserve-ci-e2e-test/isvc-init-fail-385bf1-predictor-644d7ccfd9-47sc9"
Sep 17 17:50:10.881340 ip-10-0-134-96 kubenswrapper[2574]: E0917 17:50:10.881326    2574 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-init-fail-385bf1-predictor-serving-cert: secret "isvc-init-fail-385bf1-predictor-serving-cert" not found
Sep 17 17:50:10.881415 ip-10-0-134-96 kubenswrapper[2574]: E0917 17:50:10.881401    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/64ac92d0-3d22-415c-abab-8405fb594950-proxy-tls podName:64ac92d0-3d22-415c-abab-8405fb594950 nodeName:}" failed. No retries permitted until 2026-09-17 17:50:11.381382681 +0000 UTC m=+966.570256378 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/64ac92d0-3d22-415c-abab-8405fb594950-proxy-tls") pod "isvc-init-fail-385bf1-predictor-644d7ccfd9-47sc9" (UID: "64ac92d0-3d22-415c-abab-8405fb594950") : secret "isvc-init-fail-385bf1-predictor-serving-cert" not found
Sep 17 17:50:10.881488 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:50:10.881456    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/64ac92d0-3d22-415c-abab-8405fb594950-kserve-provision-location\") pod \"isvc-init-fail-385bf1-predictor-644d7ccfd9-47sc9\" (UID: \"64ac92d0-3d22-415c-abab-8405fb594950\") " pod="kserve-ci-e2e-test/isvc-init-fail-385bf1-predictor-644d7ccfd9-47sc9"
Sep 17 17:50:10.881743 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:50:10.881726    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/64ac92d0-3d22-415c-abab-8405fb594950-cabundle-cert\") pod \"isvc-init-fail-385bf1-predictor-644d7ccfd9-47sc9\" (UID: \"64ac92d0-3d22-415c-abab-8405fb594950\") " pod="kserve-ci-e2e-test/isvc-init-fail-385bf1-predictor-644d7ccfd9-47sc9"
Sep 17 17:50:10.881796 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:50:10.881761    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/64ac92d0-3d22-415c-abab-8405fb594950-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-385bf1-predictor-644d7ccfd9-47sc9\" (UID: \"64ac92d0-3d22-415c-abab-8405fb594950\") " pod="kserve-ci-e2e-test/isvc-init-fail-385bf1-predictor-644d7ccfd9-47sc9"
Sep 17 17:50:10.892120 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:50:10.892093    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8nq8f\" (UniqueName: \"kubernetes.io/projected/64ac92d0-3d22-415c-abab-8405fb594950-kube-api-access-8nq8f\") pod \"isvc-init-fail-385bf1-predictor-644d7ccfd9-47sc9\" (UID: \"64ac92d0-3d22-415c-abab-8405fb594950\") " pod="kserve-ci-e2e-test/isvc-init-fail-385bf1-predictor-644d7ccfd9-47sc9"
Sep 17 17:50:11.091261 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:50:11.091198    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j4ds_d3f553e8-b416-4d7d-bbef-991a0404b287/konnectivity-agent/0.log" limit=1024
Sep 17 17:50:11.385684 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:50:11.385611    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/64ac92d0-3d22-415c-abab-8405fb594950-proxy-tls\") pod \"isvc-init-fail-385bf1-predictor-644d7ccfd9-47sc9\" (UID: \"64ac92d0-3d22-415c-abab-8405fb594950\") " pod="kserve-ci-e2e-test/isvc-init-fail-385bf1-predictor-644d7ccfd9-47sc9"
Sep 17 17:50:11.387908 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:50:11.387881    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/64ac92d0-3d22-415c-abab-8405fb594950-proxy-tls\") pod \"isvc-init-fail-385bf1-predictor-644d7ccfd9-47sc9\" (UID: \"64ac92d0-3d22-415c-abab-8405fb594950\") " pod="kserve-ci-e2e-test/isvc-init-fail-385bf1-predictor-644d7ccfd9-47sc9"
Sep 17 17:50:11.673858 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:50:11.673779    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-385bf1-predictor-644d7ccfd9-47sc9"
Sep 17 17:50:11.945604 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:50:11.945534    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-385bf1-predictor-644d7ccfd9-47sc9"]
Sep 17 17:50:11.948255 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:50:11.948223    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64ac92d0_3d22_415c_abab_8405fb594950.slice/crio-8cbc7f64242c74e2e476bad6bf15c4b4c1f103ef04af630751e02fe1a6bade71 WatchSource:0}: Error finding container 8cbc7f64242c74e2e476bad6bf15c4b4c1f103ef04af630751e02fe1a6bade71: Status 404 returned error can't find the container with id 8cbc7f64242c74e2e476bad6bf15c4b4c1f103ef04af630751e02fe1a6bade71
Sep 17 17:50:12.056446 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:50:12.056412    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-385bf1-predictor-644d7ccfd9-47sc9" event={"ID":"64ac92d0-3d22-415c-abab-8405fb594950","Type":"ContainerStarted","Data":"fb9124b9ba660d40e5e83447103f614e09171b5a0443a896322d5ace5fc08a06"}
Sep 17 17:50:12.056555 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:50:12.056488    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-385bf1-predictor-644d7ccfd9-47sc9" event={"ID":"64ac92d0-3d22-415c-abab-8405fb594950","Type":"ContainerStarted","Data":"8cbc7f64242c74e2e476bad6bf15c4b4c1f103ef04af630751e02fe1a6bade71"}
Sep 17 17:50:19.084768 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:50:19.084740    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-385bf1-predictor-644d7ccfd9-47sc9_64ac92d0-3d22-415c-abab-8405fb594950/storage-initializer/0.log"
Sep 17 17:50:19.085187 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:50:19.084781    2574 generic.go:356] "Generic (PLEG): container finished" podID="64ac92d0-3d22-415c-abab-8405fb594950" containerID="fb9124b9ba660d40e5e83447103f614e09171b5a0443a896322d5ace5fc08a06" exitCode=1
Sep 17 17:50:19.085187 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:50:19.084854    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-385bf1-predictor-644d7ccfd9-47sc9" event={"ID":"64ac92d0-3d22-415c-abab-8405fb594950","Type":"ContainerDied","Data":"fb9124b9ba660d40e5e83447103f614e09171b5a0443a896322d5ace5fc08a06"}
Sep 17 17:50:20.091178 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:50:20.091150    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-385bf1-predictor-644d7ccfd9-47sc9_64ac92d0-3d22-415c-abab-8405fb594950/storage-initializer/0.log"
Sep 17 17:50:20.091638 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:50:20.091235    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-385bf1-predictor-644d7ccfd9-47sc9" event={"ID":"64ac92d0-3d22-415c-abab-8405fb594950","Type":"ContainerStarted","Data":"0a9a63af77ab96dbfa578c1ebbb0c680e6835cc6c19d26d09b67b09dc0e1f321"}
Sep 17 17:50:20.719125 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:50:20.719072    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-385bf1-predictor-644d7ccfd9-47sc9"]
Sep 17 17:50:20.790111 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:50:20.790080    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j4ds_d3f553e8-b416-4d7d-bbef-991a0404b287/konnectivity-agent/0.log" limit=1024
Sep 17 17:50:21.094506 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:50:21.094468    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-init-fail-385bf1-predictor-644d7ccfd9-47sc9" podUID="64ac92d0-3d22-415c-abab-8405fb594950" containerName="storage-initializer" containerID="cri-o://0a9a63af77ab96dbfa578c1ebbb0c680e6835cc6c19d26d09b67b09dc0e1f321" gracePeriod=30
Sep 17 17:50:22.937500 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:50:22.937477    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-385bf1-predictor-644d7ccfd9-47sc9_64ac92d0-3d22-415c-abab-8405fb594950/storage-initializer/1.log"
Sep 17 17:50:22.937868 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:50:22.937854    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-385bf1-predictor-644d7ccfd9-47sc9_64ac92d0-3d22-415c-abab-8405fb594950/storage-initializer/0.log"
Sep 17 17:50:22.940219 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:50:22.940200    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-385bf1-predictor-644d7ccfd9-47sc9"
Sep 17 17:50:22.966768 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:50:22.966736    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8nq8f\" (UniqueName: \"kubernetes.io/projected/64ac92d0-3d22-415c-abab-8405fb594950-kube-api-access-8nq8f\") pod \"64ac92d0-3d22-415c-abab-8405fb594950\" (UID: \"64ac92d0-3d22-415c-abab-8405fb594950\") "
Sep 17 17:50:22.966927 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:50:22.966773    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/64ac92d0-3d22-415c-abab-8405fb594950-proxy-tls\") pod \"64ac92d0-3d22-415c-abab-8405fb594950\" (UID: \"64ac92d0-3d22-415c-abab-8405fb594950\") "
Sep 17 17:50:22.966927 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:50:22.966817    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/64ac92d0-3d22-415c-abab-8405fb594950-kserve-provision-location\") pod \"64ac92d0-3d22-415c-abab-8405fb594950\" (UID: \"64ac92d0-3d22-415c-abab-8405fb594950\") "
Sep 17 17:50:22.966927 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:50:22.966875    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/64ac92d0-3d22-415c-abab-8405fb594950-cabundle-cert\") pod \"64ac92d0-3d22-415c-abab-8405fb594950\" (UID: \"64ac92d0-3d22-415c-abab-8405fb594950\") "
Sep 17 17:50:22.966927 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:50:22.966916    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/64ac92d0-3d22-415c-abab-8405fb594950-kube-rbac-proxy-sar-config\") pod \"64ac92d0-3d22-415c-abab-8405fb594950\" (UID: \"64ac92d0-3d22-415c-abab-8405fb594950\") "
Sep 17 17:50:22.967184 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:50:22.967155    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64ac92d0-3d22-415c-abab-8405fb594950-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "64ac92d0-3d22-415c-abab-8405fb594950" (UID: "64ac92d0-3d22-415c-abab-8405fb594950"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:50:22.967256 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:50:22.967228    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64ac92d0-3d22-415c-abab-8405fb594950-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "64ac92d0-3d22-415c-abab-8405fb594950" (UID: "64ac92d0-3d22-415c-abab-8405fb594950"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:50:22.967293 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:50:22.967258    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64ac92d0-3d22-415c-abab-8405fb594950-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "64ac92d0-3d22-415c-abab-8405fb594950" (UID: "64ac92d0-3d22-415c-abab-8405fb594950"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:50:22.969023 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:50:22.968996    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64ac92d0-3d22-415c-abab-8405fb594950-kube-api-access-8nq8f" (OuterVolumeSpecName: "kube-api-access-8nq8f") pod "64ac92d0-3d22-415c-abab-8405fb594950" (UID: "64ac92d0-3d22-415c-abab-8405fb594950"). InnerVolumeSpecName "kube-api-access-8nq8f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:50:22.969165 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:50:22.969004    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/64ac92d0-3d22-415c-abab-8405fb594950-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "64ac92d0-3d22-415c-abab-8405fb594950" (UID: "64ac92d0-3d22-415c-abab-8405fb594950"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:50:23.068190 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:50:23.068158    2574 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/64ac92d0-3d22-415c-abab-8405fb594950-kserve-provision-location\") on node \"ip-10-0-134-96.ec2.internal\" DevicePath \"\""
Sep 17 17:50:23.068190 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:50:23.068187    2574 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/64ac92d0-3d22-415c-abab-8405fb594950-cabundle-cert\") on node \"ip-10-0-134-96.ec2.internal\" DevicePath \"\""
Sep 17 17:50:23.068190 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:50:23.068196    2574 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/64ac92d0-3d22-415c-abab-8405fb594950-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-96.ec2.internal\" DevicePath \"\""
Sep 17 17:50:23.068413 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:50:23.068204    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8nq8f\" (UniqueName: \"kubernetes.io/projected/64ac92d0-3d22-415c-abab-8405fb594950-kube-api-access-8nq8f\") on node \"ip-10-0-134-96.ec2.internal\" DevicePath \"\""
Sep 17 17:50:23.068413 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:50:23.068214    2574 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/64ac92d0-3d22-415c-abab-8405fb594950-proxy-tls\") on node \"ip-10-0-134-96.ec2.internal\" DevicePath \"\""
Sep 17 17:50:23.102460 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:50:23.102433    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-385bf1-predictor-644d7ccfd9-47sc9_64ac92d0-3d22-415c-abab-8405fb594950/storage-initializer/1.log"
Sep 17 17:50:23.102872 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:50:23.102856    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-385bf1-predictor-644d7ccfd9-47sc9_64ac92d0-3d22-415c-abab-8405fb594950/storage-initializer/0.log"
Sep 17 17:50:23.102941 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:50:23.102890    2574 generic.go:356] "Generic (PLEG): container finished" podID="64ac92d0-3d22-415c-abab-8405fb594950" containerID="0a9a63af77ab96dbfa578c1ebbb0c680e6835cc6c19d26d09b67b09dc0e1f321" exitCode=1
Sep 17 17:50:23.102941 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:50:23.102932    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-385bf1-predictor-644d7ccfd9-47sc9" event={"ID":"64ac92d0-3d22-415c-abab-8405fb594950","Type":"ContainerDied","Data":"0a9a63af77ab96dbfa578c1ebbb0c680e6835cc6c19d26d09b67b09dc0e1f321"}
Sep 17 17:50:23.103008 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:50:23.102950    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-385bf1-predictor-644d7ccfd9-47sc9"
Sep 17 17:50:23.103008 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:50:23.102980    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-385bf1-predictor-644d7ccfd9-47sc9" event={"ID":"64ac92d0-3d22-415c-abab-8405fb594950","Type":"ContainerDied","Data":"8cbc7f64242c74e2e476bad6bf15c4b4c1f103ef04af630751e02fe1a6bade71"}
Sep 17 17:50:23.103008 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:50:23.103005    2574 scope.go:117] "RemoveContainer" containerID="0a9a63af77ab96dbfa578c1ebbb0c680e6835cc6c19d26d09b67b09dc0e1f321"
Sep 17 17:50:23.111588 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:50:23.111570    2574 scope.go:117] "RemoveContainer" containerID="fb9124b9ba660d40e5e83447103f614e09171b5a0443a896322d5ace5fc08a06"
Sep 17 17:50:23.118686 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:50:23.118669    2574 scope.go:117] "RemoveContainer" containerID="0a9a63af77ab96dbfa578c1ebbb0c680e6835cc6c19d26d09b67b09dc0e1f321"
Sep 17 17:50:23.118922 ip-10-0-134-96 kubenswrapper[2574]: E0917 17:50:23.118903    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a9a63af77ab96dbfa578c1ebbb0c680e6835cc6c19d26d09b67b09dc0e1f321\": container with ID starting with 0a9a63af77ab96dbfa578c1ebbb0c680e6835cc6c19d26d09b67b09dc0e1f321 not found: ID does not exist" containerID="0a9a63af77ab96dbfa578c1ebbb0c680e6835cc6c19d26d09b67b09dc0e1f321"
Sep 17 17:50:23.118981 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:50:23.118933    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a9a63af77ab96dbfa578c1ebbb0c680e6835cc6c19d26d09b67b09dc0e1f321"} err="failed to get container status \"0a9a63af77ab96dbfa578c1ebbb0c680e6835cc6c19d26d09b67b09dc0e1f321\": rpc error: code = NotFound desc = could not find container \"0a9a63af77ab96dbfa578c1ebbb0c680e6835cc6c19d26d09b67b09dc0e1f321\": container with ID starting with 0a9a63af77ab96dbfa578c1ebbb0c680e6835cc6c19d26d09b67b09dc0e1f321 not found: ID does not exist"
Sep 17 17:50:23.118981 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:50:23.118954    2574 scope.go:117] "RemoveContainer" containerID="fb9124b9ba660d40e5e83447103f614e09171b5a0443a896322d5ace5fc08a06"
Sep 17 17:50:23.119195 ip-10-0-134-96 kubenswrapper[2574]: E0917 17:50:23.119176    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb9124b9ba660d40e5e83447103f614e09171b5a0443a896322d5ace5fc08a06\": container with ID starting with fb9124b9ba660d40e5e83447103f614e09171b5a0443a896322d5ace5fc08a06 not found: ID does not exist" containerID="fb9124b9ba660d40e5e83447103f614e09171b5a0443a896322d5ace5fc08a06"
Sep 17 17:50:23.119238 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:50:23.119200    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb9124b9ba660d40e5e83447103f614e09171b5a0443a896322d5ace5fc08a06"} err="failed to get container status \"fb9124b9ba660d40e5e83447103f614e09171b5a0443a896322d5ace5fc08a06\": rpc error: code = NotFound desc = could not find container \"fb9124b9ba660d40e5e83447103f614e09171b5a0443a896322d5ace5fc08a06\": container with ID starting with fb9124b9ba660d40e5e83447103f614e09171b5a0443a896322d5ace5fc08a06 not found: ID does not exist"
Sep 17 17:50:23.146919 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:50:23.146895    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-385bf1-predictor-644d7ccfd9-47sc9"]
Sep 17 17:50:23.151767 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:50:23.151746    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-385bf1-predictor-644d7ccfd9-47sc9"]
Sep 17 17:50:24.953889 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:50:24.953854    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="64ac92d0-3d22-415c-abab-8405fb594950" path="/var/lib/kubelet/pods/64ac92d0-3d22-415c-abab-8405fb594950/volumes"
Sep 17 17:50:25.950936 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:50:25.950900    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j4ds_d3f553e8-b416-4d7d-bbef-991a0404b287/konnectivity-agent/0.log" limit=1024
Sep 17 17:50:25.985531 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:50:25.985500    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-40485-predictor-578cc94c-zhjcw"]
Sep 17 17:50:25.985859 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:50:25.985772    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="64ac92d0-3d22-415c-abab-8405fb594950" containerName="storage-initializer"
Sep 17 17:50:25.985859 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:50:25.985783    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="64ac92d0-3d22-415c-abab-8405fb594950" containerName="storage-initializer"
Sep 17 17:50:25.985859 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:50:25.985798    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="64ac92d0-3d22-415c-abab-8405fb594950" containerName="storage-initializer"
Sep 17 17:50:25.985859 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:50:25.985804    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="64ac92d0-3d22-415c-abab-8405fb594950" containerName="storage-initializer"
Sep 17 17:50:25.985859 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:50:25.985851    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="64ac92d0-3d22-415c-abab-8405fb594950" containerName="storage-initializer"
Sep 17 17:50:25.985859 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:50:25.985859    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="64ac92d0-3d22-415c-abab-8405fb594950" containerName="storage-initializer"
Sep 17 17:50:25.990352 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:50:25.990337    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-40485-predictor-578cc94c-zhjcw"
Sep 17 17:50:25.993147 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:50:25.993127    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-40485-predictor-serving-cert\""
Sep 17 17:50:25.993498 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:50:25.993471    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 17:50:25.993498 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:50:25.993491    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 17:50:25.993633 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:50:25.993471    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-40485-kube-rbac-proxy-sar-config\""
Sep 17 17:50:25.993633 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:50:25.993513    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 17:50:25.993633 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:50:25.993516    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 17:50:25.993633 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:50:25.993556    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-cm7p9\""
Sep 17 17:50:26.007490 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:50:26.007464    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-40485-predictor-578cc94c-zhjcw"]
Sep 17 17:50:26.090860 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:50:26.090836    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/91ed4c8e-643f-496d-9b92-eaad8dedddd6-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-40485-predictor-578cc94c-zhjcw\" (UID: \"91ed4c8e-643f-496d-9b92-eaad8dedddd6\") " pod="kserve-ci-e2e-test/raw-sklearn-40485-predictor-578cc94c-zhjcw"
Sep 17 17:50:26.091011 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:50:26.090864    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r48vq\" (UniqueName: \"kubernetes.io/projected/91ed4c8e-643f-496d-9b92-eaad8dedddd6-kube-api-access-r48vq\") pod \"raw-sklearn-40485-predictor-578cc94c-zhjcw\" (UID: \"91ed4c8e-643f-496d-9b92-eaad8dedddd6\") " pod="kserve-ci-e2e-test/raw-sklearn-40485-predictor-578cc94c-zhjcw"
Sep 17 17:50:26.091011 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:50:26.090896    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/91ed4c8e-643f-496d-9b92-eaad8dedddd6-cabundle-cert\") pod \"raw-sklearn-40485-predictor-578cc94c-zhjcw\" (UID: \"91ed4c8e-643f-496d-9b92-eaad8dedddd6\") " pod="kserve-ci-e2e-test/raw-sklearn-40485-predictor-578cc94c-zhjcw"
Sep 17 17:50:26.091011 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:50:26.090970    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/91ed4c8e-643f-496d-9b92-eaad8dedddd6-kserve-provision-location\") pod \"raw-sklearn-40485-predictor-578cc94c-zhjcw\" (UID: \"91ed4c8e-643f-496d-9b92-eaad8dedddd6\") " pod="kserve-ci-e2e-test/raw-sklearn-40485-predictor-578cc94c-zhjcw"
Sep 17 17:50:26.091181 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:50:26.091016    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/91ed4c8e-643f-496d-9b92-eaad8dedddd6-proxy-tls\") pod \"raw-sklearn-40485-predictor-578cc94c-zhjcw\" (UID: \"91ed4c8e-643f-496d-9b92-eaad8dedddd6\") " pod="kserve-ci-e2e-test/raw-sklearn-40485-predictor-578cc94c-zhjcw"
Sep 17 17:50:26.139442 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:50:26.139399    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j4ds_d3f553e8-b416-4d7d-bbef-991a0404b287/konnectivity-agent/0.log" limit=1024
Sep 17 17:50:26.191650 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:50:26.191626    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/91ed4c8e-643f-496d-9b92-eaad8dedddd6-proxy-tls\") pod \"raw-sklearn-40485-predictor-578cc94c-zhjcw\" (UID: \"91ed4c8e-643f-496d-9b92-eaad8dedddd6\") " pod="kserve-ci-e2e-test/raw-sklearn-40485-predictor-578cc94c-zhjcw"
Sep 17 17:50:26.191766 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:50:26.191678    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/91ed4c8e-643f-496d-9b92-eaad8dedddd6-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-40485-predictor-578cc94c-zhjcw\" (UID: \"91ed4c8e-643f-496d-9b92-eaad8dedddd6\") " pod="kserve-ci-e2e-test/raw-sklearn-40485-predictor-578cc94c-zhjcw"
Sep 17 17:50:26.191766 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:50:26.191713    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r48vq\" (UniqueName: \"kubernetes.io/projected/91ed4c8e-643f-496d-9b92-eaad8dedddd6-kube-api-access-r48vq\") pod \"raw-sklearn-40485-predictor-578cc94c-zhjcw\" (UID: \"91ed4c8e-643f-496d-9b92-eaad8dedddd6\") " pod="kserve-ci-e2e-test/raw-sklearn-40485-predictor-578cc94c-zhjcw"
Sep 17 17:50:26.191766 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:50:26.191734    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/91ed4c8e-643f-496d-9b92-eaad8dedddd6-cabundle-cert\") pod \"raw-sklearn-40485-predictor-578cc94c-zhjcw\" (UID: \"91ed4c8e-643f-496d-9b92-eaad8dedddd6\") " pod="kserve-ci-e2e-test/raw-sklearn-40485-predictor-578cc94c-zhjcw"
Sep 17 17:50:26.191766 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:50:26.191761    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/91ed4c8e-643f-496d-9b92-eaad8dedddd6-kserve-provision-location\") pod \"raw-sklearn-40485-predictor-578cc94c-zhjcw\" (UID: \"91ed4c8e-643f-496d-9b92-eaad8dedddd6\") " pod="kserve-ci-e2e-test/raw-sklearn-40485-predictor-578cc94c-zhjcw"
Sep 17 17:50:26.192194 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:50:26.192175    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/91ed4c8e-643f-496d-9b92-eaad8dedddd6-kserve-provision-location\") pod \"raw-sklearn-40485-predictor-578cc94c-zhjcw\" (UID: \"91ed4c8e-643f-496d-9b92-eaad8dedddd6\") " pod="kserve-ci-e2e-test/raw-sklearn-40485-predictor-578cc94c-zhjcw"
Sep 17 17:50:26.192437 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:50:26.192357    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/91ed4c8e-643f-496d-9b92-eaad8dedddd6-cabundle-cert\") pod \"raw-sklearn-40485-predictor-578cc94c-zhjcw\" (UID: \"91ed4c8e-643f-496d-9b92-eaad8dedddd6\") " pod="kserve-ci-e2e-test/raw-sklearn-40485-predictor-578cc94c-zhjcw"
Sep 17 17:50:26.192437 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:50:26.192391    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/91ed4c8e-643f-496d-9b92-eaad8dedddd6-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-40485-predictor-578cc94c-zhjcw\" (UID: \"91ed4c8e-643f-496d-9b92-eaad8dedddd6\") " pod="kserve-ci-e2e-test/raw-sklearn-40485-predictor-578cc94c-zhjcw"
Sep 17 17:50:26.194129 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:50:26.194112    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/91ed4c8e-643f-496d-9b92-eaad8dedddd6-proxy-tls\") pod \"raw-sklearn-40485-predictor-578cc94c-zhjcw\" (UID: \"91ed4c8e-643f-496d-9b92-eaad8dedddd6\") " pod="kserve-ci-e2e-test/raw-sklearn-40485-predictor-578cc94c-zhjcw"
Sep 17 17:50:26.200400 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:50:26.200379    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r48vq\" (UniqueName: \"kubernetes.io/projected/91ed4c8e-643f-496d-9b92-eaad8dedddd6-kube-api-access-r48vq\") pod \"raw-sklearn-40485-predictor-578cc94c-zhjcw\" (UID: \"91ed4c8e-643f-496d-9b92-eaad8dedddd6\") " pod="kserve-ci-e2e-test/raw-sklearn-40485-predictor-578cc94c-zhjcw"
Sep 17 17:50:26.299728 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:50:26.299709    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-40485-predictor-578cc94c-zhjcw"
Sep 17 17:50:26.568599 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:50:26.568576    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-40485-predictor-578cc94c-zhjcw"]
Sep 17 17:50:26.570986 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:50:26.570959    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91ed4c8e_643f_496d_9b92_eaad8dedddd6.slice/crio-a09eeb0b5697502a4457846981c45dfb841a47cdfedbc7abca84f0c094bf7c80 WatchSource:0}: Error finding container a09eeb0b5697502a4457846981c45dfb841a47cdfedbc7abca84f0c094bf7c80: Status 404 returned error can't find the container with id a09eeb0b5697502a4457846981c45dfb841a47cdfedbc7abca84f0c094bf7c80
Sep 17 17:50:27.120522 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:50:27.120322    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-40485-predictor-578cc94c-zhjcw" event={"ID":"91ed4c8e-643f-496d-9b92-eaad8dedddd6","Type":"ContainerStarted","Data":"04ae1aa3765e2031ca8e8921ccef4e24f89d1100f808bc579bbe07ea75cf16af"}
Sep 17 17:50:27.120522 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:50:27.120401    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-40485-predictor-578cc94c-zhjcw" event={"ID":"91ed4c8e-643f-496d-9b92-eaad8dedddd6","Type":"ContainerStarted","Data":"a09eeb0b5697502a4457846981c45dfb841a47cdfedbc7abca84f0c094bf7c80"}
Sep 17 17:50:31.138461 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:50:31.138424    2574 generic.go:356] "Generic (PLEG): container finished" podID="91ed4c8e-643f-496d-9b92-eaad8dedddd6" containerID="04ae1aa3765e2031ca8e8921ccef4e24f89d1100f808bc579bbe07ea75cf16af" exitCode=0
Sep 17 17:50:31.138854 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:50:31.138499    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-40485-predictor-578cc94c-zhjcw" event={"ID":"91ed4c8e-643f-496d-9b92-eaad8dedddd6","Type":"ContainerDied","Data":"04ae1aa3765e2031ca8e8921ccef4e24f89d1100f808bc579bbe07ea75cf16af"}
Sep 17 17:50:31.786021 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:50:31.785990    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j4ds_d3f553e8-b416-4d7d-bbef-991a0404b287/konnectivity-agent/0.log" limit=1024
Sep 17 17:50:31.926072 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:50:31.926023    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j4ds_d3f553e8-b416-4d7d-bbef-991a0404b287/konnectivity-agent/0.log" limit=1024
Sep 17 17:50:32.034360 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:50:32.034328    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j4ds_d3f553e8-b416-4d7d-bbef-991a0404b287/konnectivity-agent/0.log" limit=1024
Sep 17 17:50:32.144387 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:50:32.144309    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-40485-predictor-578cc94c-zhjcw" event={"ID":"91ed4c8e-643f-496d-9b92-eaad8dedddd6","Type":"ContainerStarted","Data":"af57939ec3b08504c8449b7ed19f5d1a9c0b7381ac8797f165a212a8ad159185"}
Sep 17 17:50:32.144387 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:50:32.144375    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-40485-predictor-578cc94c-zhjcw" event={"ID":"91ed4c8e-643f-496d-9b92-eaad8dedddd6","Type":"ContainerStarted","Data":"ae565604daf581c3327f1c3bd86506a9f14ae28f582b9a2f1c53e8458401e892"}
Sep 17 17:50:32.144803 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:50:32.144561    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-40485-predictor-578cc94c-zhjcw"
Sep 17 17:50:32.170541 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:50:32.170493    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/raw-sklearn-40485-predictor-578cc94c-zhjcw" podStartSLOduration=3.384347173 podStartE2EDuration="7.170480278s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.786133105s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:50:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:50:32.169345714 +0000 UTC m=+987.358219420" watchObservedRunningTime="2026-09-17 17:50:32.170480278 +0000 UTC m=+987.359353983"
Sep 17 17:50:32.199642 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:50:32.199611    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j4ds_d3f553e8-b416-4d7d-bbef-991a0404b287/konnectivity-agent/0.log" limit=1024
Sep 17 17:50:33.148201 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:50:33.148172    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-40485-predictor-578cc94c-zhjcw"
Sep 17 17:50:33.149378 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:50:33.149349    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-40485-predictor-578cc94c-zhjcw" podUID="91ed4c8e-643f-496d-9b92-eaad8dedddd6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Sep 17 17:50:34.152089 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:50:34.152025    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-40485-predictor-578cc94c-zhjcw" podUID="91ed4c8e-643f-496d-9b92-eaad8dedddd6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Sep 17 17:50:39.157454 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:50:39.157418    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-40485-predictor-578cc94c-zhjcw"
Sep 17 17:50:39.157896 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:50:39.157870    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-40485-predictor-578cc94c-zhjcw" podUID="91ed4c8e-643f-496d-9b92-eaad8dedddd6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Sep 17 17:50:49.157914 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:50:49.157871    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-40485-predictor-578cc94c-zhjcw" podUID="91ed4c8e-643f-496d-9b92-eaad8dedddd6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Sep 17 17:50:59.158837 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:50:59.158800    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-40485-predictor-578cc94c-zhjcw" podUID="91ed4c8e-643f-496d-9b92-eaad8dedddd6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Sep 17 17:51:09.158801 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:51:09.158765    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-40485-predictor-578cc94c-zhjcw" podUID="91ed4c8e-643f-496d-9b92-eaad8dedddd6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Sep 17 17:51:19.158388 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:51:19.158348    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-40485-predictor-578cc94c-zhjcw" podUID="91ed4c8e-643f-496d-9b92-eaad8dedddd6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Sep 17 17:51:29.158456 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:51:29.158420    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-40485-predictor-578cc94c-zhjcw" podUID="91ed4c8e-643f-496d-9b92-eaad8dedddd6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Sep 17 17:51:39.158225 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:51:39.158191    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-40485-predictor-578cc94c-zhjcw"
Sep 17 17:51:46.082344 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:51:46.082309    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-40485-predictor-578cc94c-zhjcw"]
Sep 17 17:51:46.082728 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:51:46.082553    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-40485-predictor-578cc94c-zhjcw" podUID="91ed4c8e-643f-496d-9b92-eaad8dedddd6" containerName="kserve-container" containerID="cri-o://ae565604daf581c3327f1c3bd86506a9f14ae28f582b9a2f1c53e8458401e892" gracePeriod=30
Sep 17 17:51:46.082728 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:51:46.082593    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-40485-predictor-578cc94c-zhjcw" podUID="91ed4c8e-643f-496d-9b92-eaad8dedddd6" containerName="kube-rbac-proxy" containerID="cri-o://af57939ec3b08504c8449b7ed19f5d1a9c0b7381ac8797f165a212a8ad159185" gracePeriod=30
Sep 17 17:51:46.227504 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:51:46.227473    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j4ds_d3f553e8-b416-4d7d-bbef-991a0404b287/konnectivity-agent/0.log" limit=1024
Sep 17 17:51:46.437774 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:51:46.437709    2574 generic.go:356] "Generic (PLEG): container finished" podID="91ed4c8e-643f-496d-9b92-eaad8dedddd6" containerID="af57939ec3b08504c8449b7ed19f5d1a9c0b7381ac8797f165a212a8ad159185" exitCode=2
Sep 17 17:51:46.437774 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:51:46.437753    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-40485-predictor-578cc94c-zhjcw" event={"ID":"91ed4c8e-643f-496d-9b92-eaad8dedddd6","Type":"ContainerDied","Data":"af57939ec3b08504c8449b7ed19f5d1a9c0b7381ac8797f165a212a8ad159185"}
Sep 17 17:51:49.152905 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:51:49.152823    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-40485-predictor-578cc94c-zhjcw" podUID="91ed4c8e-643f-496d-9b92-eaad8dedddd6" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.32:8643/healthz\": dial tcp 10.134.0.32:8643: connect: connection refused"
Sep 17 17:51:49.158142 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:51:49.158118    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-40485-predictor-578cc94c-zhjcw" podUID="91ed4c8e-643f-496d-9b92-eaad8dedddd6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Sep 17 17:51:50.114137 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:51:50.114114    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-40485-predictor-578cc94c-zhjcw"
Sep 17 17:51:50.238410 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:51:50.238388    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/91ed4c8e-643f-496d-9b92-eaad8dedddd6-kube-rbac-proxy-sar-config\") pod \"91ed4c8e-643f-496d-9b92-eaad8dedddd6\" (UID: \"91ed4c8e-643f-496d-9b92-eaad8dedddd6\") "
Sep 17 17:51:50.238812 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:51:50.238436    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/91ed4c8e-643f-496d-9b92-eaad8dedddd6-proxy-tls\") pod \"91ed4c8e-643f-496d-9b92-eaad8dedddd6\" (UID: \"91ed4c8e-643f-496d-9b92-eaad8dedddd6\") "
Sep 17 17:51:50.238812 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:51:50.238455    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/91ed4c8e-643f-496d-9b92-eaad8dedddd6-cabundle-cert\") pod \"91ed4c8e-643f-496d-9b92-eaad8dedddd6\" (UID: \"91ed4c8e-643f-496d-9b92-eaad8dedddd6\") "
Sep 17 17:51:50.238812 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:51:50.238477    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/91ed4c8e-643f-496d-9b92-eaad8dedddd6-kserve-provision-location\") pod \"91ed4c8e-643f-496d-9b92-eaad8dedddd6\" (UID: \"91ed4c8e-643f-496d-9b92-eaad8dedddd6\") "
Sep 17 17:51:50.238812 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:51:50.238662    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r48vq\" (UniqueName: \"kubernetes.io/projected/91ed4c8e-643f-496d-9b92-eaad8dedddd6-kube-api-access-r48vq\") pod \"91ed4c8e-643f-496d-9b92-eaad8dedddd6\" (UID: \"91ed4c8e-643f-496d-9b92-eaad8dedddd6\") "
Sep 17 17:51:50.238812 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:51:50.238756    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91ed4c8e-643f-496d-9b92-eaad8dedddd6-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "91ed4c8e-643f-496d-9b92-eaad8dedddd6" (UID: "91ed4c8e-643f-496d-9b92-eaad8dedddd6"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:51:50.239004 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:51:50.238810    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91ed4c8e-643f-496d-9b92-eaad8dedddd6-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "91ed4c8e-643f-496d-9b92-eaad8dedddd6" (UID: "91ed4c8e-643f-496d-9b92-eaad8dedddd6"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:51:50.239004 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:51:50.238825    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91ed4c8e-643f-496d-9b92-eaad8dedddd6-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "91ed4c8e-643f-496d-9b92-eaad8dedddd6" (UID: "91ed4c8e-643f-496d-9b92-eaad8dedddd6"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:51:50.239004 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:51:50.238935    2574 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/91ed4c8e-643f-496d-9b92-eaad8dedddd6-kserve-provision-location\") on node \"ip-10-0-134-96.ec2.internal\" DevicePath \"\""
Sep 17 17:51:50.239004 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:51:50.238949    2574 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/91ed4c8e-643f-496d-9b92-eaad8dedddd6-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-96.ec2.internal\" DevicePath \"\""
Sep 17 17:51:50.239004 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:51:50.238957    2574 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/91ed4c8e-643f-496d-9b92-eaad8dedddd6-cabundle-cert\") on node \"ip-10-0-134-96.ec2.internal\" DevicePath \"\""
Sep 17 17:51:50.240470 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:51:50.240450    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/91ed4c8e-643f-496d-9b92-eaad8dedddd6-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "91ed4c8e-643f-496d-9b92-eaad8dedddd6" (UID: "91ed4c8e-643f-496d-9b92-eaad8dedddd6"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:51:50.240612 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:51:50.240591    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91ed4c8e-643f-496d-9b92-eaad8dedddd6-kube-api-access-r48vq" (OuterVolumeSpecName: "kube-api-access-r48vq") pod "91ed4c8e-643f-496d-9b92-eaad8dedddd6" (UID: "91ed4c8e-643f-496d-9b92-eaad8dedddd6"). InnerVolumeSpecName "kube-api-access-r48vq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:51:50.339917 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:51:50.339894    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r48vq\" (UniqueName: \"kubernetes.io/projected/91ed4c8e-643f-496d-9b92-eaad8dedddd6-kube-api-access-r48vq\") on node \"ip-10-0-134-96.ec2.internal\" DevicePath \"\""
Sep 17 17:51:50.339917 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:51:50.339915    2574 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/91ed4c8e-643f-496d-9b92-eaad8dedddd6-proxy-tls\") on node \"ip-10-0-134-96.ec2.internal\" DevicePath \"\""
Sep 17 17:51:50.455767 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:51:50.455740    2574 generic.go:356] "Generic (PLEG): container finished" podID="91ed4c8e-643f-496d-9b92-eaad8dedddd6" containerID="ae565604daf581c3327f1c3bd86506a9f14ae28f582b9a2f1c53e8458401e892" exitCode=0
Sep 17 17:51:50.455888 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:51:50.455806    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-40485-predictor-578cc94c-zhjcw"
Sep 17 17:51:50.455888 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:51:50.455822    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-40485-predictor-578cc94c-zhjcw" event={"ID":"91ed4c8e-643f-496d-9b92-eaad8dedddd6","Type":"ContainerDied","Data":"ae565604daf581c3327f1c3bd86506a9f14ae28f582b9a2f1c53e8458401e892"}
Sep 17 17:51:50.456002 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:51:50.455907    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-40485-predictor-578cc94c-zhjcw" event={"ID":"91ed4c8e-643f-496d-9b92-eaad8dedddd6","Type":"ContainerDied","Data":"a09eeb0b5697502a4457846981c45dfb841a47cdfedbc7abca84f0c094bf7c80"}
Sep 17 17:51:50.456002 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:51:50.455934    2574 scope.go:117] "RemoveContainer" containerID="af57939ec3b08504c8449b7ed19f5d1a9c0b7381ac8797f165a212a8ad159185"
Sep 17 17:51:50.464346 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:51:50.464325    2574 scope.go:117] "RemoveContainer" containerID="ae565604daf581c3327f1c3bd86506a9f14ae28f582b9a2f1c53e8458401e892"
Sep 17 17:51:50.471101 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:51:50.471062    2574 scope.go:117] "RemoveContainer" containerID="04ae1aa3765e2031ca8e8921ccef4e24f89d1100f808bc579bbe07ea75cf16af"
Sep 17 17:51:50.477621 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:51:50.477604    2574 scope.go:117] "RemoveContainer" containerID="af57939ec3b08504c8449b7ed19f5d1a9c0b7381ac8797f165a212a8ad159185"
Sep 17 17:51:50.477870 ip-10-0-134-96 kubenswrapper[2574]: E0917 17:51:50.477850    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af57939ec3b08504c8449b7ed19f5d1a9c0b7381ac8797f165a212a8ad159185\": container with ID starting with af57939ec3b08504c8449b7ed19f5d1a9c0b7381ac8797f165a212a8ad159185 not found: ID does not exist" containerID="af57939ec3b08504c8449b7ed19f5d1a9c0b7381ac8797f165a212a8ad159185"
Sep 17 17:51:50.477909 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:51:50.477877    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af57939ec3b08504c8449b7ed19f5d1a9c0b7381ac8797f165a212a8ad159185"} err="failed to get container status \"af57939ec3b08504c8449b7ed19f5d1a9c0b7381ac8797f165a212a8ad159185\": rpc error: code = NotFound desc = could not find container \"af57939ec3b08504c8449b7ed19f5d1a9c0b7381ac8797f165a212a8ad159185\": container with ID starting with af57939ec3b08504c8449b7ed19f5d1a9c0b7381ac8797f165a212a8ad159185 not found: ID does not exist"
Sep 17 17:51:50.477909 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:51:50.477893    2574 scope.go:117] "RemoveContainer" containerID="ae565604daf581c3327f1c3bd86506a9f14ae28f582b9a2f1c53e8458401e892"
Sep 17 17:51:50.478113 ip-10-0-134-96 kubenswrapper[2574]: E0917 17:51:50.478098    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae565604daf581c3327f1c3bd86506a9f14ae28f582b9a2f1c53e8458401e892\": container with ID starting with ae565604daf581c3327f1c3bd86506a9f14ae28f582b9a2f1c53e8458401e892 not found: ID does not exist" containerID="ae565604daf581c3327f1c3bd86506a9f14ae28f582b9a2f1c53e8458401e892"
Sep 17 17:51:50.478151 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:51:50.478120    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae565604daf581c3327f1c3bd86506a9f14ae28f582b9a2f1c53e8458401e892"} err="failed to get container status \"ae565604daf581c3327f1c3bd86506a9f14ae28f582b9a2f1c53e8458401e892\": rpc error: code = NotFound desc = could not find container \"ae565604daf581c3327f1c3bd86506a9f14ae28f582b9a2f1c53e8458401e892\": container with ID starting with ae565604daf581c3327f1c3bd86506a9f14ae28f582b9a2f1c53e8458401e892 not found: ID does not exist"
Sep 17 17:51:50.478151 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:51:50.478132    2574 scope.go:117] "RemoveContainer" containerID="04ae1aa3765e2031ca8e8921ccef4e24f89d1100f808bc579bbe07ea75cf16af"
Sep 17 17:51:50.478344 ip-10-0-134-96 kubenswrapper[2574]: E0917 17:51:50.478328    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"04ae1aa3765e2031ca8e8921ccef4e24f89d1100f808bc579bbe07ea75cf16af\": container with ID starting with 04ae1aa3765e2031ca8e8921ccef4e24f89d1100f808bc579bbe07ea75cf16af not found: ID does not exist" containerID="04ae1aa3765e2031ca8e8921ccef4e24f89d1100f808bc579bbe07ea75cf16af"
Sep 17 17:51:50.478394 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:51:50.478347    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04ae1aa3765e2031ca8e8921ccef4e24f89d1100f808bc579bbe07ea75cf16af"} err="failed to get container status \"04ae1aa3765e2031ca8e8921ccef4e24f89d1100f808bc579bbe07ea75cf16af\": rpc error: code = NotFound desc = could not find container \"04ae1aa3765e2031ca8e8921ccef4e24f89d1100f808bc579bbe07ea75cf16af\": container with ID starting with 04ae1aa3765e2031ca8e8921ccef4e24f89d1100f808bc579bbe07ea75cf16af not found: ID does not exist"
Sep 17 17:51:50.489826 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:51:50.489781    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-40485-predictor-578cc94c-zhjcw"]
Sep 17 17:51:50.499087 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:51:50.499068    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-40485-predictor-578cc94c-zhjcw"]
Sep 17 17:51:50.953498 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:51:50.953423    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="91ed4c8e-643f-496d-9b92-eaad8dedddd6" path="/var/lib/kubelet/pods/91ed4c8e-643f-496d-9b92-eaad8dedddd6/volumes"
Sep 17 17:51:51.354845 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:51:51.354819    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j4ds_d3f553e8-b416-4d7d-bbef-991a0404b287/konnectivity-agent/0.log" limit=1024
Sep 17 17:51:51.406141 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:51:51.406111    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-de90d-predictor-6bff856c59-ctgp5"]
Sep 17 17:51:51.406399 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:51:51.406387    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="91ed4c8e-643f-496d-9b92-eaad8dedddd6" containerName="storage-initializer"
Sep 17 17:51:51.406399 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:51:51.406400    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="91ed4c8e-643f-496d-9b92-eaad8dedddd6" containerName="storage-initializer"
Sep 17 17:51:51.406480 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:51:51.406408    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="91ed4c8e-643f-496d-9b92-eaad8dedddd6" containerName="kserve-container"
Sep 17 17:51:51.406480 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:51:51.406414    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="91ed4c8e-643f-496d-9b92-eaad8dedddd6" containerName="kserve-container"
Sep 17 17:51:51.406480 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:51:51.406424    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="91ed4c8e-643f-496d-9b92-eaad8dedddd6" containerName="kube-rbac-proxy"
Sep 17 17:51:51.406480 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:51:51.406430    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="91ed4c8e-643f-496d-9b92-eaad8dedddd6" containerName="kube-rbac-proxy"
Sep 17 17:51:51.406480 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:51:51.406474    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="91ed4c8e-643f-496d-9b92-eaad8dedddd6" containerName="kserve-container"
Sep 17 17:51:51.406649 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:51:51.406485    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="91ed4c8e-643f-496d-9b92-eaad8dedddd6" containerName="kube-rbac-proxy"
Sep 17 17:51:51.410877 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:51:51.410860    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-de90d-predictor-6bff856c59-ctgp5"
Sep 17 17:51:51.415865 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:51:51.415843    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-runtime-de90d-predictor-serving-cert\""
Sep 17 17:51:51.416197 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:51:51.416179    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 17:51:51.416318 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:51:51.416234    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 17:51:51.416318 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:51:51.416243    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-runtime-de90d-kube-rbac-proxy-sar-config\""
Sep 17 17:51:51.416453 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:51:51.416327    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 17:51:51.417368 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:51:51.417349    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 17:51:51.417460 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:51:51.417352    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-cm7p9\""
Sep 17 17:51:51.430629 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:51:51.430609    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-de90d-predictor-6bff856c59-ctgp5"]
Sep 17 17:51:51.549082 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:51:51.549058    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/baf80c31-78c2-41cf-9adf-858361747a08-cabundle-cert\") pod \"raw-sklearn-runtime-de90d-predictor-6bff856c59-ctgp5\" (UID: \"baf80c31-78c2-41cf-9adf-858361747a08\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-de90d-predictor-6bff856c59-ctgp5"
Sep 17 17:51:51.549192 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:51:51.549095    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9jf8h\" (UniqueName: \"kubernetes.io/projected/baf80c31-78c2-41cf-9adf-858361747a08-kube-api-access-9jf8h\") pod \"raw-sklearn-runtime-de90d-predictor-6bff856c59-ctgp5\" (UID: \"baf80c31-78c2-41cf-9adf-858361747a08\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-de90d-predictor-6bff856c59-ctgp5"
Sep 17 17:51:51.549192 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:51:51.549182    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/baf80c31-78c2-41cf-9adf-858361747a08-kserve-provision-location\") pod \"raw-sklearn-runtime-de90d-predictor-6bff856c59-ctgp5\" (UID: \"baf80c31-78c2-41cf-9adf-858361747a08\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-de90d-predictor-6bff856c59-ctgp5"
Sep 17 17:51:51.549263 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:51:51.549220    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/baf80c31-78c2-41cf-9adf-858361747a08-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-runtime-de90d-predictor-6bff856c59-ctgp5\" (UID: \"baf80c31-78c2-41cf-9adf-858361747a08\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-de90d-predictor-6bff856c59-ctgp5"
Sep 17 17:51:51.549263 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:51:51.549251    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/baf80c31-78c2-41cf-9adf-858361747a08-proxy-tls\") pod \"raw-sklearn-runtime-de90d-predictor-6bff856c59-ctgp5\" (UID: \"baf80c31-78c2-41cf-9adf-858361747a08\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-de90d-predictor-6bff856c59-ctgp5"
Sep 17 17:51:51.649787 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:51:51.649730    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/baf80c31-78c2-41cf-9adf-858361747a08-kserve-provision-location\") pod \"raw-sklearn-runtime-de90d-predictor-6bff856c59-ctgp5\" (UID: \"baf80c31-78c2-41cf-9adf-858361747a08\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-de90d-predictor-6bff856c59-ctgp5"
Sep 17 17:51:51.649787 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:51:51.649772    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/baf80c31-78c2-41cf-9adf-858361747a08-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-runtime-de90d-predictor-6bff856c59-ctgp5\" (UID: \"baf80c31-78c2-41cf-9adf-858361747a08\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-de90d-predictor-6bff856c59-ctgp5"
Sep 17 17:51:51.649957 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:51:51.649792    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/baf80c31-78c2-41cf-9adf-858361747a08-proxy-tls\") pod \"raw-sklearn-runtime-de90d-predictor-6bff856c59-ctgp5\" (UID: \"baf80c31-78c2-41cf-9adf-858361747a08\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-de90d-predictor-6bff856c59-ctgp5"
Sep 17 17:51:51.649957 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:51:51.649810    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/baf80c31-78c2-41cf-9adf-858361747a08-cabundle-cert\") pod \"raw-sklearn-runtime-de90d-predictor-6bff856c59-ctgp5\" (UID: \"baf80c31-78c2-41cf-9adf-858361747a08\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-de90d-predictor-6bff856c59-ctgp5"
Sep 17 17:51:51.649957 ip-10-0-134-96 kubenswrapper[2574]: E0917 17:51:51.649908    2574 secret.go:189] Couldn't get secret kserve-ci-e2e-test/raw-sklearn-runtime-de90d-predictor-serving-cert: secret "raw-sklearn-runtime-de90d-predictor-serving-cert" not found
Sep 17 17:51:51.649957 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:51:51.649921    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9jf8h\" (UniqueName: \"kubernetes.io/projected/baf80c31-78c2-41cf-9adf-858361747a08-kube-api-access-9jf8h\") pod \"raw-sklearn-runtime-de90d-predictor-6bff856c59-ctgp5\" (UID: \"baf80c31-78c2-41cf-9adf-858361747a08\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-de90d-predictor-6bff856c59-ctgp5"
Sep 17 17:51:51.650199 ip-10-0-134-96 kubenswrapper[2574]: E0917 17:51:51.649973    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/baf80c31-78c2-41cf-9adf-858361747a08-proxy-tls podName:baf80c31-78c2-41cf-9adf-858361747a08 nodeName:}" failed. No retries permitted until 2026-09-17 17:51:52.149954088 +0000 UTC m=+1067.338827792 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/baf80c31-78c2-41cf-9adf-858361747a08-proxy-tls") pod "raw-sklearn-runtime-de90d-predictor-6bff856c59-ctgp5" (UID: "baf80c31-78c2-41cf-9adf-858361747a08") : secret "raw-sklearn-runtime-de90d-predictor-serving-cert" not found
Sep 17 17:51:51.650199 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:51:51.650117    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/baf80c31-78c2-41cf-9adf-858361747a08-kserve-provision-location\") pod \"raw-sklearn-runtime-de90d-predictor-6bff856c59-ctgp5\" (UID: \"baf80c31-78c2-41cf-9adf-858361747a08\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-de90d-predictor-6bff856c59-ctgp5"
Sep 17 17:51:51.650416 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:51:51.650400    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/baf80c31-78c2-41cf-9adf-858361747a08-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-runtime-de90d-predictor-6bff856c59-ctgp5\" (UID: \"baf80c31-78c2-41cf-9adf-858361747a08\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-de90d-predictor-6bff856c59-ctgp5"
Sep 17 17:51:51.650515 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:51:51.650499    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/baf80c31-78c2-41cf-9adf-858361747a08-cabundle-cert\") pod \"raw-sklearn-runtime-de90d-predictor-6bff856c59-ctgp5\" (UID: \"baf80c31-78c2-41cf-9adf-858361747a08\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-de90d-predictor-6bff856c59-ctgp5"
Sep 17 17:51:51.659103 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:51:51.659080    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9jf8h\" (UniqueName: \"kubernetes.io/projected/baf80c31-78c2-41cf-9adf-858361747a08-kube-api-access-9jf8h\") pod \"raw-sklearn-runtime-de90d-predictor-6bff856c59-ctgp5\" (UID: \"baf80c31-78c2-41cf-9adf-858361747a08\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-de90d-predictor-6bff856c59-ctgp5"
Sep 17 17:51:51.851405 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:51:51.851379    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j4ds_d3f553e8-b416-4d7d-bbef-991a0404b287/konnectivity-agent/0.log" limit=1024
Sep 17 17:51:52.153678 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:51:52.153650    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/baf80c31-78c2-41cf-9adf-858361747a08-proxy-tls\") pod \"raw-sklearn-runtime-de90d-predictor-6bff856c59-ctgp5\" (UID: \"baf80c31-78c2-41cf-9adf-858361747a08\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-de90d-predictor-6bff856c59-ctgp5"
Sep 17 17:51:52.156076 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:51:52.156034    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/baf80c31-78c2-41cf-9adf-858361747a08-proxy-tls\") pod \"raw-sklearn-runtime-de90d-predictor-6bff856c59-ctgp5\" (UID: \"baf80c31-78c2-41cf-9adf-858361747a08\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-de90d-predictor-6bff856c59-ctgp5"
Sep 17 17:51:52.321056 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:51:52.321006    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-de90d-predictor-6bff856c59-ctgp5"
Sep 17 17:51:52.592709 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:51:52.592683    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-de90d-predictor-6bff856c59-ctgp5"]
Sep 17 17:51:52.595147 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:51:52.595120    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbaf80c31_78c2_41cf_9adf_858361747a08.slice/crio-2c118874ec69e7772fbe48a834f822b036feda77d720b635c1685b9dfc5af43d WatchSource:0}: Error finding container 2c118874ec69e7772fbe48a834f822b036feda77d720b635c1685b9dfc5af43d: Status 404 returned error can't find the container with id 2c118874ec69e7772fbe48a834f822b036feda77d720b635c1685b9dfc5af43d
Sep 17 17:51:53.470198 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:51:53.470163    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-de90d-predictor-6bff856c59-ctgp5" event={"ID":"baf80c31-78c2-41cf-9adf-858361747a08","Type":"ContainerStarted","Data":"376d8b41553f08a2cd52cb92c939410e54f9776e8357c894b1e02ec9e81e14b2"}
Sep 17 17:51:53.470370 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:51:53.470222    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-de90d-predictor-6bff856c59-ctgp5" event={"ID":"baf80c31-78c2-41cf-9adf-858361747a08","Type":"ContainerStarted","Data":"2c118874ec69e7772fbe48a834f822b036feda77d720b635c1685b9dfc5af43d"}
Sep 17 17:51:57.487451 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:51:57.487420    2574 generic.go:356] "Generic (PLEG): container finished" podID="baf80c31-78c2-41cf-9adf-858361747a08" containerID="376d8b41553f08a2cd52cb92c939410e54f9776e8357c894b1e02ec9e81e14b2" exitCode=0
Sep 17 17:51:57.487825 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:51:57.487467    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-de90d-predictor-6bff856c59-ctgp5" event={"ID":"baf80c31-78c2-41cf-9adf-858361747a08","Type":"ContainerDied","Data":"376d8b41553f08a2cd52cb92c939410e54f9776e8357c894b1e02ec9e81e14b2"}
Sep 17 17:51:58.493374 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:51:58.493344    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-de90d-predictor-6bff856c59-ctgp5" event={"ID":"baf80c31-78c2-41cf-9adf-858361747a08","Type":"ContainerStarted","Data":"ae721e0e22616fba903de8a7870d0a72459809a8a92a57232cbc4f3809529a86"}
Sep 17 17:51:58.493713 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:51:58.493400    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-de90d-predictor-6bff856c59-ctgp5" event={"ID":"baf80c31-78c2-41cf-9adf-858361747a08","Type":"ContainerStarted","Data":"fda293185e48edad2bb176fef474bf5768ef847a22e5994b7452f34c3a7e1db0"}
Sep 17 17:51:58.493713 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:51:58.493578    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-de90d-predictor-6bff856c59-ctgp5"
Sep 17 17:51:58.515768 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:51:58.515714    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/raw-sklearn-runtime-de90d-predictor-6bff856c59-ctgp5" podStartSLOduration=3.654006175 podStartE2EDuration="7.515697801s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.861691626s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:51:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:51:58.514912811 +0000 UTC m=+1073.703786527" watchObservedRunningTime="2026-09-17 17:51:58.515697801 +0000 UTC m=+1073.704571507"
Sep 17 17:51:59.497715 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:51:59.497685    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-de90d-predictor-6bff856c59-ctgp5"
Sep 17 17:51:59.498884 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:51:59.498855    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-de90d-predictor-6bff856c59-ctgp5" podUID="baf80c31-78c2-41cf-9adf-858361747a08" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Sep 17 17:52:00.501160 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:52:00.501111    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-de90d-predictor-6bff856c59-ctgp5" podUID="baf80c31-78c2-41cf-9adf-858361747a08" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Sep 17 17:52:05.505709 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:52:05.505684    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-de90d-predictor-6bff856c59-ctgp5"
Sep 17 17:52:05.506156 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:52:05.506131    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-de90d-predictor-6bff856c59-ctgp5" podUID="baf80c31-78c2-41cf-9adf-858361747a08" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Sep 17 17:52:10.615369 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:52:10.615317    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j4ds_d3f553e8-b416-4d7d-bbef-991a0404b287/konnectivity-agent/0.log" limit=1024
Sep 17 17:52:15.506775 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:52:15.506740    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-de90d-predictor-6bff856c59-ctgp5" podUID="baf80c31-78c2-41cf-9adf-858361747a08" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Sep 17 17:52:25.506978 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:52:25.506929    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-de90d-predictor-6bff856c59-ctgp5" podUID="baf80c31-78c2-41cf-9adf-858361747a08" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Sep 17 17:52:35.506764 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:52:35.506727    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-de90d-predictor-6bff856c59-ctgp5" podUID="baf80c31-78c2-41cf-9adf-858361747a08" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Sep 17 17:52:45.507001 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:52:45.506964    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-de90d-predictor-6bff856c59-ctgp5" podUID="baf80c31-78c2-41cf-9adf-858361747a08" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Sep 17 17:52:55.506151 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:52:55.506113    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-de90d-predictor-6bff856c59-ctgp5" podUID="baf80c31-78c2-41cf-9adf-858361747a08" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Sep 17 17:53:05.507183 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:53:05.507155    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-de90d-predictor-6bff856c59-ctgp5"
Sep 17 17:53:06.654807 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:53:06.654754    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j4ds_d3f553e8-b416-4d7d-bbef-991a0404b287/konnectivity-agent/0.log" limit=1024
Sep 17 17:53:11.449244 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:53:11.449212    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-de90d-predictor-6bff856c59-ctgp5"]
Sep 17 17:53:11.449599 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:53:11.449439    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-runtime-de90d-predictor-6bff856c59-ctgp5" podUID="baf80c31-78c2-41cf-9adf-858361747a08" containerName="kserve-container" containerID="cri-o://fda293185e48edad2bb176fef474bf5768ef847a22e5994b7452f34c3a7e1db0" gracePeriod=30
Sep 17 17:53:11.449599 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:53:11.449497    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-runtime-de90d-predictor-6bff856c59-ctgp5" podUID="baf80c31-78c2-41cf-9adf-858361747a08" containerName="kube-rbac-proxy" containerID="cri-o://ae721e0e22616fba903de8a7870d0a72459809a8a92a57232cbc4f3809529a86" gracePeriod=30
Sep 17 17:53:11.476029 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:53:11.476008    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j4ds_d3f553e8-b416-4d7d-bbef-991a0404b287/konnectivity-agent/0.log" limit=1024
Sep 17 17:53:11.782889 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:53:11.782860    2574 generic.go:356] "Generic (PLEG): container finished" podID="baf80c31-78c2-41cf-9adf-858361747a08" containerID="ae721e0e22616fba903de8a7870d0a72459809a8a92a57232cbc4f3809529a86" exitCode=2
Sep 17 17:53:11.783065 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:53:11.782928    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-de90d-predictor-6bff856c59-ctgp5" event={"ID":"baf80c31-78c2-41cf-9adf-858361747a08","Type":"ContainerDied","Data":"ae721e0e22616fba903de8a7870d0a72459809a8a92a57232cbc4f3809529a86"}
Sep 17 17:53:12.009016 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:53:12.008970    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j4ds_d3f553e8-b416-4d7d-bbef-991a0404b287/konnectivity-agent/0.log" limit=1024
Sep 17 17:53:15.501885 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:53:15.501842    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-de90d-predictor-6bff856c59-ctgp5" podUID="baf80c31-78c2-41cf-9adf-858361747a08" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.33:8643/healthz\": dial tcp 10.134.0.33:8643: connect: connection refused"
Sep 17 17:53:15.506326 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:53:15.506300    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-de90d-predictor-6bff856c59-ctgp5" podUID="baf80c31-78c2-41cf-9adf-858361747a08" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Sep 17 17:53:15.587891 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:53:15.587870    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-de90d-predictor-6bff856c59-ctgp5"
Sep 17 17:53:15.724651 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:53:15.724584    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9jf8h\" (UniqueName: \"kubernetes.io/projected/baf80c31-78c2-41cf-9adf-858361747a08-kube-api-access-9jf8h\") pod \"baf80c31-78c2-41cf-9adf-858361747a08\" (UID: \"baf80c31-78c2-41cf-9adf-858361747a08\") "
Sep 17 17:53:15.724651 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:53:15.724619    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/baf80c31-78c2-41cf-9adf-858361747a08-proxy-tls\") pod \"baf80c31-78c2-41cf-9adf-858361747a08\" (UID: \"baf80c31-78c2-41cf-9adf-858361747a08\") "
Sep 17 17:53:15.724651 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:53:15.724642    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/baf80c31-78c2-41cf-9adf-858361747a08-kube-rbac-proxy-sar-config\") pod \"baf80c31-78c2-41cf-9adf-858361747a08\" (UID: \"baf80c31-78c2-41cf-9adf-858361747a08\") "
Sep 17 17:53:15.724855 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:53:15.724663    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/baf80c31-78c2-41cf-9adf-858361747a08-cabundle-cert\") pod \"baf80c31-78c2-41cf-9adf-858361747a08\" (UID: \"baf80c31-78c2-41cf-9adf-858361747a08\") "
Sep 17 17:53:15.724855 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:53:15.724708    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/baf80c31-78c2-41cf-9adf-858361747a08-kserve-provision-location\") pod \"baf80c31-78c2-41cf-9adf-858361747a08\" (UID: \"baf80c31-78c2-41cf-9adf-858361747a08\") "
Sep 17 17:53:15.725097 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:53:15.725073    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/baf80c31-78c2-41cf-9adf-858361747a08-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "baf80c31-78c2-41cf-9adf-858361747a08" (UID: "baf80c31-78c2-41cf-9adf-858361747a08"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:53:15.725097 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:53:15.725085    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/baf80c31-78c2-41cf-9adf-858361747a08-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "baf80c31-78c2-41cf-9adf-858361747a08" (UID: "baf80c31-78c2-41cf-9adf-858361747a08"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:53:15.725253 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:53:15.725097    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/baf80c31-78c2-41cf-9adf-858361747a08-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "baf80c31-78c2-41cf-9adf-858361747a08" (UID: "baf80c31-78c2-41cf-9adf-858361747a08"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:53:15.726711 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:53:15.726692    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/baf80c31-78c2-41cf-9adf-858361747a08-kube-api-access-9jf8h" (OuterVolumeSpecName: "kube-api-access-9jf8h") pod "baf80c31-78c2-41cf-9adf-858361747a08" (UID: "baf80c31-78c2-41cf-9adf-858361747a08"). InnerVolumeSpecName "kube-api-access-9jf8h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:53:15.726837 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:53:15.726691    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/baf80c31-78c2-41cf-9adf-858361747a08-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "baf80c31-78c2-41cf-9adf-858361747a08" (UID: "baf80c31-78c2-41cf-9adf-858361747a08"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:53:15.799192 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:53:15.799166    2574 generic.go:356] "Generic (PLEG): container finished" podID="baf80c31-78c2-41cf-9adf-858361747a08" containerID="fda293185e48edad2bb176fef474bf5768ef847a22e5994b7452f34c3a7e1db0" exitCode=0
Sep 17 17:53:15.799290 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:53:15.799224    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-de90d-predictor-6bff856c59-ctgp5"
Sep 17 17:53:15.799290 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:53:15.799244    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-de90d-predictor-6bff856c59-ctgp5" event={"ID":"baf80c31-78c2-41cf-9adf-858361747a08","Type":"ContainerDied","Data":"fda293185e48edad2bb176fef474bf5768ef847a22e5994b7452f34c3a7e1db0"}
Sep 17 17:53:15.799365 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:53:15.799309    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-de90d-predictor-6bff856c59-ctgp5" event={"ID":"baf80c31-78c2-41cf-9adf-858361747a08","Type":"ContainerDied","Data":"2c118874ec69e7772fbe48a834f822b036feda77d720b635c1685b9dfc5af43d"}
Sep 17 17:53:15.799365 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:53:15.799340    2574 scope.go:117] "RemoveContainer" containerID="ae721e0e22616fba903de8a7870d0a72459809a8a92a57232cbc4f3809529a86"
Sep 17 17:53:15.807161 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:53:15.807146    2574 scope.go:117] "RemoveContainer" containerID="fda293185e48edad2bb176fef474bf5768ef847a22e5994b7452f34c3a7e1db0"
Sep 17 17:53:15.814535 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:53:15.814520    2574 scope.go:117] "RemoveContainer" containerID="376d8b41553f08a2cd52cb92c939410e54f9776e8357c894b1e02ec9e81e14b2"
Sep 17 17:53:15.821317 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:53:15.821294    2574 scope.go:117] "RemoveContainer" containerID="ae721e0e22616fba903de8a7870d0a72459809a8a92a57232cbc4f3809529a86"
Sep 17 17:53:15.821913 ip-10-0-134-96 kubenswrapper[2574]: E0917 17:53:15.821888    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae721e0e22616fba903de8a7870d0a72459809a8a92a57232cbc4f3809529a86\": container with ID starting with ae721e0e22616fba903de8a7870d0a72459809a8a92a57232cbc4f3809529a86 not found: ID does not exist" containerID="ae721e0e22616fba903de8a7870d0a72459809a8a92a57232cbc4f3809529a86"
Sep 17 17:53:15.821997 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:53:15.821921    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae721e0e22616fba903de8a7870d0a72459809a8a92a57232cbc4f3809529a86"} err="failed to get container status \"ae721e0e22616fba903de8a7870d0a72459809a8a92a57232cbc4f3809529a86\": rpc error: code = NotFound desc = could not find container \"ae721e0e22616fba903de8a7870d0a72459809a8a92a57232cbc4f3809529a86\": container with ID starting with ae721e0e22616fba903de8a7870d0a72459809a8a92a57232cbc4f3809529a86 not found: ID does not exist"
Sep 17 17:53:15.821997 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:53:15.821942    2574 scope.go:117] "RemoveContainer" containerID="fda293185e48edad2bb176fef474bf5768ef847a22e5994b7452f34c3a7e1db0"
Sep 17 17:53:15.822245 ip-10-0-134-96 kubenswrapper[2574]: E0917 17:53:15.822228    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fda293185e48edad2bb176fef474bf5768ef847a22e5994b7452f34c3a7e1db0\": container with ID starting with fda293185e48edad2bb176fef474bf5768ef847a22e5994b7452f34c3a7e1db0 not found: ID does not exist" containerID="fda293185e48edad2bb176fef474bf5768ef847a22e5994b7452f34c3a7e1db0"
Sep 17 17:53:15.822297 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:53:15.822251    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fda293185e48edad2bb176fef474bf5768ef847a22e5994b7452f34c3a7e1db0"} err="failed to get container status \"fda293185e48edad2bb176fef474bf5768ef847a22e5994b7452f34c3a7e1db0\": rpc error: code = NotFound desc = could not find container \"fda293185e48edad2bb176fef474bf5768ef847a22e5994b7452f34c3a7e1db0\": container with ID starting with fda293185e48edad2bb176fef474bf5768ef847a22e5994b7452f34c3a7e1db0 not found: ID does not exist"
Sep 17 17:53:15.822297 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:53:15.822267    2574 scope.go:117] "RemoveContainer" containerID="376d8b41553f08a2cd52cb92c939410e54f9776e8357c894b1e02ec9e81e14b2"
Sep 17 17:53:15.822553 ip-10-0-134-96 kubenswrapper[2574]: E0917 17:53:15.822536    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"376d8b41553f08a2cd52cb92c939410e54f9776e8357c894b1e02ec9e81e14b2\": container with ID starting with 376d8b41553f08a2cd52cb92c939410e54f9776e8357c894b1e02ec9e81e14b2 not found: ID does not exist" containerID="376d8b41553f08a2cd52cb92c939410e54f9776e8357c894b1e02ec9e81e14b2"
Sep 17 17:53:15.822611 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:53:15.822562    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"376d8b41553f08a2cd52cb92c939410e54f9776e8357c894b1e02ec9e81e14b2"} err="failed to get container status \"376d8b41553f08a2cd52cb92c939410e54f9776e8357c894b1e02ec9e81e14b2\": rpc error: code = NotFound desc = could not find container \"376d8b41553f08a2cd52cb92c939410e54f9776e8357c894b1e02ec9e81e14b2\": container with ID starting with 376d8b41553f08a2cd52cb92c939410e54f9776e8357c894b1e02ec9e81e14b2 not found: ID does not exist"
Sep 17 17:53:15.823540 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:53:15.823523    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-de90d-predictor-6bff856c59-ctgp5"]
Sep 17 17:53:15.825813 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:53:15.825795    2574 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/baf80c31-78c2-41cf-9adf-858361747a08-proxy-tls\") on node \"ip-10-0-134-96.ec2.internal\" DevicePath \"\""
Sep 17 17:53:15.825813 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:53:15.825813    2574 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/baf80c31-78c2-41cf-9adf-858361747a08-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-96.ec2.internal\" DevicePath \"\""
Sep 17 17:53:15.826166 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:53:15.825822    2574 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/baf80c31-78c2-41cf-9adf-858361747a08-cabundle-cert\") on node \"ip-10-0-134-96.ec2.internal\" DevicePath \"\""
Sep 17 17:53:15.826166 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:53:15.825830    2574 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/baf80c31-78c2-41cf-9adf-858361747a08-kserve-provision-location\") on node \"ip-10-0-134-96.ec2.internal\" DevicePath \"\""
Sep 17 17:53:15.826166 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:53:15.825838    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9jf8h\" (UniqueName: \"kubernetes.io/projected/baf80c31-78c2-41cf-9adf-858361747a08-kube-api-access-9jf8h\") on node \"ip-10-0-134-96.ec2.internal\" DevicePath \"\""
Sep 17 17:53:15.827809 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:53:15.827789    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-de90d-predictor-6bff856c59-ctgp5"]
Sep 17 17:53:16.589969 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:53:16.589923    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j4ds_d3f553e8-b416-4d7d-bbef-991a0404b287/konnectivity-agent/0.log" limit=1024
Sep 17 17:53:16.663015 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:53:16.662986    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-7d998654b-d2jqt"]
Sep 17 17:53:16.663307 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:53:16.663294    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="baf80c31-78c2-41cf-9adf-858361747a08" containerName="kube-rbac-proxy"
Sep 17 17:53:16.663355 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:53:16.663310    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="baf80c31-78c2-41cf-9adf-858361747a08" containerName="kube-rbac-proxy"
Sep 17 17:53:16.663355 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:53:16.663322    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="baf80c31-78c2-41cf-9adf-858361747a08" containerName="storage-initializer"
Sep 17 17:53:16.663355 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:53:16.663327    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="baf80c31-78c2-41cf-9adf-858361747a08" containerName="storage-initializer"
Sep 17 17:53:16.663355 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:53:16.663341    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="baf80c31-78c2-41cf-9adf-858361747a08" containerName="kserve-container"
Sep 17 17:53:16.663355 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:53:16.663347    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="baf80c31-78c2-41cf-9adf-858361747a08" containerName="kserve-container"
Sep 17 17:53:16.663519 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:53:16.663391    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="baf80c31-78c2-41cf-9adf-858361747a08" containerName="kserve-container"
Sep 17 17:53:16.663519 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:53:16.663399    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="baf80c31-78c2-41cf-9adf-858361747a08" containerName="kube-rbac-proxy"
Sep 17 17:53:16.667784 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:53:16.667768    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d998654b-d2jqt"
Sep 17 17:53:16.670585 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:53:16.670557    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 17:53:16.670585 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:53:16.670586    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-cm7p9\""
Sep 17 17:53:16.670585 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:53:16.670557    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-predictor-serving-cert\""
Sep 17 17:53:16.670857 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:53:16.670565    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 17:53:16.670857 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:53:16.670563    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-kube-rbac-proxy-sar-config\""
Sep 17 17:53:16.671337 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:53:16.671320    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j4ds_d3f553e8-b416-4d7d-bbef-991a0404b287/konnectivity-agent/0.log" limit=1024
Sep 17 17:53:16.671544 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:53:16.671531    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 17:53:16.671762 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:53:16.671747    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 17:53:16.676633 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:53:16.676613    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-7d998654b-d2jqt"]
Sep 17 17:53:16.822202 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:53:16.822171    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j4ds_d3f553e8-b416-4d7d-bbef-991a0404b287/konnectivity-agent/0.log" limit=1024
Sep 17 17:53:16.834301 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:53:16.834278    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8c6366fb-8881-4eb3-b237-cd268fa59e26-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-7d998654b-d2jqt\" (UID: \"8c6366fb-8881-4eb3-b237-cd268fa59e26\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d998654b-d2jqt"
Sep 17 17:53:16.834381 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:53:16.834321    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8x7cs\" (UniqueName: \"kubernetes.io/projected/8c6366fb-8881-4eb3-b237-cd268fa59e26-kube-api-access-8x7cs\") pod \"isvc-canary-create-predictor-7d998654b-d2jqt\" (UID: \"8c6366fb-8881-4eb3-b237-cd268fa59e26\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d998654b-d2jqt"
Sep 17 17:53:16.834381 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:53:16.834346    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8c6366fb-8881-4eb3-b237-cd268fa59e26-proxy-tls\") pod \"isvc-canary-create-predictor-7d998654b-d2jqt\" (UID: \"8c6366fb-8881-4eb3-b237-cd268fa59e26\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d998654b-d2jqt"
Sep 17 17:53:16.834458 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:53:16.834412    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8c6366fb-8881-4eb3-b237-cd268fa59e26-cabundle-cert\") pod \"isvc-canary-create-predictor-7d998654b-d2jqt\" (UID: \"8c6366fb-8881-4eb3-b237-cd268fa59e26\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d998654b-d2jqt"
Sep 17 17:53:16.834458 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:53:16.834452    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8c6366fb-8881-4eb3-b237-cd268fa59e26-kserve-provision-location\") pod \"isvc-canary-create-predictor-7d998654b-d2jqt\" (UID: \"8c6366fb-8881-4eb3-b237-cd268fa59e26\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d998654b-d2jqt"
Sep 17 17:53:16.935573 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:53:16.935499    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8c6366fb-8881-4eb3-b237-cd268fa59e26-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-7d998654b-d2jqt\" (UID: \"8c6366fb-8881-4eb3-b237-cd268fa59e26\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d998654b-d2jqt"
Sep 17 17:53:16.935573 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:53:16.935549    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8x7cs\" (UniqueName: \"kubernetes.io/projected/8c6366fb-8881-4eb3-b237-cd268fa59e26-kube-api-access-8x7cs\") pod \"isvc-canary-create-predictor-7d998654b-d2jqt\" (UID: \"8c6366fb-8881-4eb3-b237-cd268fa59e26\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d998654b-d2jqt"
Sep 17 17:53:16.935759 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:53:16.935586    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8c6366fb-8881-4eb3-b237-cd268fa59e26-proxy-tls\") pod \"isvc-canary-create-predictor-7d998654b-d2jqt\" (UID: \"8c6366fb-8881-4eb3-b237-cd268fa59e26\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d998654b-d2jqt"
Sep 17 17:53:16.935759 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:53:16.935620    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8c6366fb-8881-4eb3-b237-cd268fa59e26-cabundle-cert\") pod \"isvc-canary-create-predictor-7d998654b-d2jqt\" (UID: \"8c6366fb-8881-4eb3-b237-cd268fa59e26\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d998654b-d2jqt"
Sep 17 17:53:16.935759 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:53:16.935667    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8c6366fb-8881-4eb3-b237-cd268fa59e26-kserve-provision-location\") pod \"isvc-canary-create-predictor-7d998654b-d2jqt\" (UID: \"8c6366fb-8881-4eb3-b237-cd268fa59e26\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d998654b-d2jqt"
Sep 17 17:53:16.935759 ip-10-0-134-96 kubenswrapper[2574]: E0917 17:53:16.935744    2574 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-create-predictor-serving-cert: secret "isvc-canary-create-predictor-serving-cert" not found
Sep 17 17:53:16.935958 ip-10-0-134-96 kubenswrapper[2574]: E0917 17:53:16.935828    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8c6366fb-8881-4eb3-b237-cd268fa59e26-proxy-tls podName:8c6366fb-8881-4eb3-b237-cd268fa59e26 nodeName:}" failed. No retries permitted until 2026-09-17 17:53:17.435811314 +0000 UTC m=+1152.624685007 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/8c6366fb-8881-4eb3-b237-cd268fa59e26-proxy-tls") pod "isvc-canary-create-predictor-7d998654b-d2jqt" (UID: "8c6366fb-8881-4eb3-b237-cd268fa59e26") : secret "isvc-canary-create-predictor-serving-cert" not found
Sep 17 17:53:16.936082 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:53:16.936038    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8c6366fb-8881-4eb3-b237-cd268fa59e26-kserve-provision-location\") pod \"isvc-canary-create-predictor-7d998654b-d2jqt\" (UID: \"8c6366fb-8881-4eb3-b237-cd268fa59e26\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d998654b-d2jqt"
Sep 17 17:53:16.936356 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:53:16.936325    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8c6366fb-8881-4eb3-b237-cd268fa59e26-cabundle-cert\") pod \"isvc-canary-create-predictor-7d998654b-d2jqt\" (UID: \"8c6366fb-8881-4eb3-b237-cd268fa59e26\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d998654b-d2jqt"
Sep 17 17:53:16.936397 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:53:16.936325    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8c6366fb-8881-4eb3-b237-cd268fa59e26-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-7d998654b-d2jqt\" (UID: \"8c6366fb-8881-4eb3-b237-cd268fa59e26\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d998654b-d2jqt"
Sep 17 17:53:16.946401 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:53:16.946381    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8x7cs\" (UniqueName: \"kubernetes.io/projected/8c6366fb-8881-4eb3-b237-cd268fa59e26-kube-api-access-8x7cs\") pod \"isvc-canary-create-predictor-7d998654b-d2jqt\" (UID: \"8c6366fb-8881-4eb3-b237-cd268fa59e26\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d998654b-d2jqt"
Sep 17 17:53:16.952882 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:53:16.952860    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="baf80c31-78c2-41cf-9adf-858361747a08" path="/var/lib/kubelet/pods/baf80c31-78c2-41cf-9adf-858361747a08/volumes"
Sep 17 17:53:17.078855 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:53:17.078826    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j4ds_d3f553e8-b416-4d7d-bbef-991a0404b287/konnectivity-agent/0.log" limit=1024
Sep 17 17:53:17.439053 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:53:17.439015    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8c6366fb-8881-4eb3-b237-cd268fa59e26-proxy-tls\") pod \"isvc-canary-create-predictor-7d998654b-d2jqt\" (UID: \"8c6366fb-8881-4eb3-b237-cd268fa59e26\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d998654b-d2jqt"
Sep 17 17:53:17.441400 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:53:17.441371    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8c6366fb-8881-4eb3-b237-cd268fa59e26-proxy-tls\") pod \"isvc-canary-create-predictor-7d998654b-d2jqt\" (UID: \"8c6366fb-8881-4eb3-b237-cd268fa59e26\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d998654b-d2jqt"
Sep 17 17:53:17.577777 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:53:17.577748    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d998654b-d2jqt"
Sep 17 17:53:17.849088 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:53:17.849065    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-7d998654b-d2jqt"]
Sep 17 17:53:17.851462 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:53:17.851438    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c6366fb_8881_4eb3_b237_cd268fa59e26.slice/crio-4af367e119ce30f6a5b1eb7db28ca3ff6afefa61e87eb12e58c5b2e2480d80db WatchSource:0}: Error finding container 4af367e119ce30f6a5b1eb7db28ca3ff6afefa61e87eb12e58c5b2e2480d80db: Status 404 returned error can't find the container with id 4af367e119ce30f6a5b1eb7db28ca3ff6afefa61e87eb12e58c5b2e2480d80db
Sep 17 17:53:18.814901 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:53:18.814866    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d998654b-d2jqt" event={"ID":"8c6366fb-8881-4eb3-b237-cd268fa59e26","Type":"ContainerStarted","Data":"93bab7f54dab7c45daedeec983fb0b4bb6b098353f835c45cced24e2d009026d"}
Sep 17 17:53:18.815082 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:53:18.814920    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d998654b-d2jqt" event={"ID":"8c6366fb-8881-4eb3-b237-cd268fa59e26","Type":"ContainerStarted","Data":"4af367e119ce30f6a5b1eb7db28ca3ff6afefa61e87eb12e58c5b2e2480d80db"}
Sep 17 17:53:21.828296 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:53:21.828263    2574 generic.go:356] "Generic (PLEG): container finished" podID="8c6366fb-8881-4eb3-b237-cd268fa59e26" containerID="93bab7f54dab7c45daedeec983fb0b4bb6b098353f835c45cced24e2d009026d" exitCode=0
Sep 17 17:53:21.828689 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:53:21.828334    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d998654b-d2jqt" event={"ID":"8c6366fb-8881-4eb3-b237-cd268fa59e26","Type":"ContainerDied","Data":"93bab7f54dab7c45daedeec983fb0b4bb6b098353f835c45cced24e2d009026d"}
Sep 17 17:53:22.135445 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:53:22.135349    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j4ds_d3f553e8-b416-4d7d-bbef-991a0404b287/konnectivity-agent/0.log" limit=1024
Sep 17 17:53:22.834554 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:53:22.834523    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d998654b-d2jqt" event={"ID":"8c6366fb-8881-4eb3-b237-cd268fa59e26","Type":"ContainerStarted","Data":"3e82edc90629f5fb06bfc8303d4e6eeac17d578e000a26b4055b4c970631f722"}
Sep 17 17:53:22.834990 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:53:22.834582    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d998654b-d2jqt" event={"ID":"8c6366fb-8881-4eb3-b237-cd268fa59e26","Type":"ContainerStarted","Data":"1d532ce770a46e26a9a5f71b142003624ccb9e4deea9e5e0a3e1e628fe3f80b0"}
Sep 17 17:53:22.834990 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:53:22.834840    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d998654b-d2jqt"
Sep 17 17:53:22.835149 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:53:22.834987    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d998654b-d2jqt"
Sep 17 17:53:22.836035 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:53:22.836012    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d998654b-d2jqt" podUID="8c6366fb-8881-4eb3-b237-cd268fa59e26" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 17 17:53:22.855172 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:53:22.855125    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d998654b-d2jqt" podStartSLOduration=3.167450707 podStartE2EDuration="6.855113039s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.687662332s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:53:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:53:22.854643273 +0000 UTC m=+1158.043517000" watchObservedRunningTime="2026-09-17 17:53:22.855113039 +0000 UTC m=+1158.043986735"
Sep 17 17:53:23.838727 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:53:23.838689    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d998654b-d2jqt" podUID="8c6366fb-8881-4eb3-b237-cd268fa59e26" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 17 17:53:28.842537 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:53:28.842506    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d998654b-d2jqt"
Sep 17 17:53:28.842975 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:53:28.842950    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d998654b-d2jqt" podUID="8c6366fb-8881-4eb3-b237-cd268fa59e26" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 17 17:53:38.843609 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:53:38.843569    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d998654b-d2jqt" podUID="8c6366fb-8881-4eb3-b237-cd268fa59e26" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 17 17:53:48.843010 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:53:48.842974    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d998654b-d2jqt" podUID="8c6366fb-8881-4eb3-b237-cd268fa59e26" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 17 17:53:58.843412 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:53:58.843377    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d998654b-d2jqt" podUID="8c6366fb-8881-4eb3-b237-cd268fa59e26" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 17 17:54:06.886718 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:54:06.886692    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-45dp8_bd527736-9920-4c79-9069-1eb8be07f869/ovn-acl-logging/0.log"
Sep 17 17:54:06.887125 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:54:06.886861    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-45dp8_bd527736-9920-4c79-9069-1eb8be07f869/ovn-acl-logging/0.log"
Sep 17 17:54:08.843498 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:54:08.843456    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d998654b-d2jqt" podUID="8c6366fb-8881-4eb3-b237-cd268fa59e26" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 17 17:54:18.843139 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:54:18.843098    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d998654b-d2jqt" podUID="8c6366fb-8881-4eb3-b237-cd268fa59e26" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 17 17:54:28.842922 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:54:28.842882    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d998654b-d2jqt" podUID="8c6366fb-8881-4eb3-b237-cd268fa59e26" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 17 17:54:38.844265 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:54:38.844231    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d998654b-d2jqt"
Sep 17 17:54:46.759994 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:54:46.759961    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-7d998654b-d2jqt"]
Sep 17 17:54:46.760497 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:54:46.760317    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d998654b-d2jqt" podUID="8c6366fb-8881-4eb3-b237-cd268fa59e26" containerName="kserve-container" containerID="cri-o://1d532ce770a46e26a9a5f71b142003624ccb9e4deea9e5e0a3e1e628fe3f80b0" gracePeriod=30
Sep 17 17:54:46.760497 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:54:46.760323    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d998654b-d2jqt" podUID="8c6366fb-8881-4eb3-b237-cd268fa59e26" containerName="kube-rbac-proxy" containerID="cri-o://3e82edc90629f5fb06bfc8303d4e6eeac17d578e000a26b4055b4c970631f722" gracePeriod=30
Sep 17 17:54:46.774682 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:54:46.774662    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j4ds_d3f553e8-b416-4d7d-bbef-991a0404b287/konnectivity-agent/0.log" limit=1024
Sep 17 17:54:46.848777 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:54:46.848754    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j4ds_d3f553e8-b416-4d7d-bbef-991a0404b287/konnectivity-agent/0.log" limit=1024
Sep 17 17:54:47.152124 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:54:47.152032    2574 generic.go:356] "Generic (PLEG): container finished" podID="8c6366fb-8881-4eb3-b237-cd268fa59e26" containerID="3e82edc90629f5fb06bfc8303d4e6eeac17d578e000a26b4055b4c970631f722" exitCode=2
Sep 17 17:54:47.152124 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:54:47.152096    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d998654b-d2jqt" event={"ID":"8c6366fb-8881-4eb3-b237-cd268fa59e26","Type":"ContainerDied","Data":"3e82edc90629f5fb06bfc8303d4e6eeac17d578e000a26b4055b4c970631f722"}
Sep 17 17:54:48.839485 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:54:48.839441    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d998654b-d2jqt" podUID="8c6366fb-8881-4eb3-b237-cd268fa59e26" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.34:8643/healthz\": dial tcp 10.134.0.34:8643: connect: connection refused"
Sep 17 17:54:48.843578 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:54:48.843549    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d998654b-d2jqt" podUID="8c6366fb-8881-4eb3-b237-cd268fa59e26" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 17 17:54:51.003491 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:54:51.003457    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d998654b-d2jqt"
Sep 17 17:54:51.048306 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:54:51.048227    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8c6366fb-8881-4eb3-b237-cd268fa59e26-kube-rbac-proxy-sar-config\") pod \"8c6366fb-8881-4eb3-b237-cd268fa59e26\" (UID: \"8c6366fb-8881-4eb3-b237-cd268fa59e26\") "
Sep 17 17:54:51.048306 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:54:51.048274    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8c6366fb-8881-4eb3-b237-cd268fa59e26-cabundle-cert\") pod \"8c6366fb-8881-4eb3-b237-cd268fa59e26\" (UID: \"8c6366fb-8881-4eb3-b237-cd268fa59e26\") "
Sep 17 17:54:51.048524 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:54:51.048314    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8x7cs\" (UniqueName: \"kubernetes.io/projected/8c6366fb-8881-4eb3-b237-cd268fa59e26-kube-api-access-8x7cs\") pod \"8c6366fb-8881-4eb3-b237-cd268fa59e26\" (UID: \"8c6366fb-8881-4eb3-b237-cd268fa59e26\") "
Sep 17 17:54:51.048524 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:54:51.048377    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8c6366fb-8881-4eb3-b237-cd268fa59e26-proxy-tls\") pod \"8c6366fb-8881-4eb3-b237-cd268fa59e26\" (UID: \"8c6366fb-8881-4eb3-b237-cd268fa59e26\") "
Sep 17 17:54:51.048524 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:54:51.048400    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8c6366fb-8881-4eb3-b237-cd268fa59e26-kserve-provision-location\") pod \"8c6366fb-8881-4eb3-b237-cd268fa59e26\" (UID: \"8c6366fb-8881-4eb3-b237-cd268fa59e26\") "
Sep 17 17:54:51.048673 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:54:51.048613    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c6366fb-8881-4eb3-b237-cd268fa59e26-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "8c6366fb-8881-4eb3-b237-cd268fa59e26" (UID: "8c6366fb-8881-4eb3-b237-cd268fa59e26"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:54:51.048793 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:54:51.048760    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c6366fb-8881-4eb3-b237-cd268fa59e26-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "8c6366fb-8881-4eb3-b237-cd268fa59e26" (UID: "8c6366fb-8881-4eb3-b237-cd268fa59e26"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:54:51.048866 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:54:51.048789    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c6366fb-8881-4eb3-b237-cd268fa59e26-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "8c6366fb-8881-4eb3-b237-cd268fa59e26" (UID: "8c6366fb-8881-4eb3-b237-cd268fa59e26"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:54:51.050412 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:54:51.050386    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c6366fb-8881-4eb3-b237-cd268fa59e26-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "8c6366fb-8881-4eb3-b237-cd268fa59e26" (UID: "8c6366fb-8881-4eb3-b237-cd268fa59e26"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:54:51.050518 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:54:51.050435    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c6366fb-8881-4eb3-b237-cd268fa59e26-kube-api-access-8x7cs" (OuterVolumeSpecName: "kube-api-access-8x7cs") pod "8c6366fb-8881-4eb3-b237-cd268fa59e26" (UID: "8c6366fb-8881-4eb3-b237-cd268fa59e26"). InnerVolumeSpecName "kube-api-access-8x7cs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:54:51.149723 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:54:51.149697    2574 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8c6366fb-8881-4eb3-b237-cd268fa59e26-cabundle-cert\") on node \"ip-10-0-134-96.ec2.internal\" DevicePath \"\""
Sep 17 17:54:51.149723 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:54:51.149720    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8x7cs\" (UniqueName: \"kubernetes.io/projected/8c6366fb-8881-4eb3-b237-cd268fa59e26-kube-api-access-8x7cs\") on node \"ip-10-0-134-96.ec2.internal\" DevicePath \"\""
Sep 17 17:54:51.149723 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:54:51.149729    2574 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8c6366fb-8881-4eb3-b237-cd268fa59e26-proxy-tls\") on node \"ip-10-0-134-96.ec2.internal\" DevicePath \"\""
Sep 17 17:54:51.149932 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:54:51.149740    2574 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8c6366fb-8881-4eb3-b237-cd268fa59e26-kserve-provision-location\") on node \"ip-10-0-134-96.ec2.internal\" DevicePath \"\""
Sep 17 17:54:51.149932 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:54:51.149748    2574 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8c6366fb-8881-4eb3-b237-cd268fa59e26-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-96.ec2.internal\" DevicePath \"\""
Sep 17 17:54:51.168782 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:54:51.168753    2574 generic.go:356] "Generic (PLEG): container finished" podID="8c6366fb-8881-4eb3-b237-cd268fa59e26" containerID="1d532ce770a46e26a9a5f71b142003624ccb9e4deea9e5e0a3e1e628fe3f80b0" exitCode=0
Sep 17 17:54:51.168904 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:54:51.168809    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d998654b-d2jqt"
Sep 17 17:54:51.168904 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:54:51.168838    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d998654b-d2jqt" event={"ID":"8c6366fb-8881-4eb3-b237-cd268fa59e26","Type":"ContainerDied","Data":"1d532ce770a46e26a9a5f71b142003624ccb9e4deea9e5e0a3e1e628fe3f80b0"}
Sep 17 17:54:51.168978 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:54:51.168909    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d998654b-d2jqt" event={"ID":"8c6366fb-8881-4eb3-b237-cd268fa59e26","Type":"ContainerDied","Data":"4af367e119ce30f6a5b1eb7db28ca3ff6afefa61e87eb12e58c5b2e2480d80db"}
Sep 17 17:54:51.168978 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:54:51.168947    2574 scope.go:117] "RemoveContainer" containerID="3e82edc90629f5fb06bfc8303d4e6eeac17d578e000a26b4055b4c970631f722"
Sep 17 17:54:51.177459 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:54:51.177442    2574 scope.go:117] "RemoveContainer" containerID="1d532ce770a46e26a9a5f71b142003624ccb9e4deea9e5e0a3e1e628fe3f80b0"
Sep 17 17:54:51.184626 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:54:51.184606    2574 scope.go:117] "RemoveContainer" containerID="93bab7f54dab7c45daedeec983fb0b4bb6b098353f835c45cced24e2d009026d"
Sep 17 17:54:51.191539 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:54:51.191523    2574 scope.go:117] "RemoveContainer" containerID="3e82edc90629f5fb06bfc8303d4e6eeac17d578e000a26b4055b4c970631f722"
Sep 17 17:54:51.191796 ip-10-0-134-96 kubenswrapper[2574]: E0917 17:54:51.191778    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e82edc90629f5fb06bfc8303d4e6eeac17d578e000a26b4055b4c970631f722\": container with ID starting with 3e82edc90629f5fb06bfc8303d4e6eeac17d578e000a26b4055b4c970631f722 not found: ID does not exist" containerID="3e82edc90629f5fb06bfc8303d4e6eeac17d578e000a26b4055b4c970631f722"
Sep 17 17:54:51.191860 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:54:51.191810    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e82edc90629f5fb06bfc8303d4e6eeac17d578e000a26b4055b4c970631f722"} err="failed to get container status \"3e82edc90629f5fb06bfc8303d4e6eeac17d578e000a26b4055b4c970631f722\": rpc error: code = NotFound desc = could not find container \"3e82edc90629f5fb06bfc8303d4e6eeac17d578e000a26b4055b4c970631f722\": container with ID starting with 3e82edc90629f5fb06bfc8303d4e6eeac17d578e000a26b4055b4c970631f722 not found: ID does not exist"
Sep 17 17:54:51.191860 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:54:51.191827    2574 scope.go:117] "RemoveContainer" containerID="1d532ce770a46e26a9a5f71b142003624ccb9e4deea9e5e0a3e1e628fe3f80b0"
Sep 17 17:54:51.192054 ip-10-0-134-96 kubenswrapper[2574]: E0917 17:54:51.192027    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d532ce770a46e26a9a5f71b142003624ccb9e4deea9e5e0a3e1e628fe3f80b0\": container with ID starting with 1d532ce770a46e26a9a5f71b142003624ccb9e4deea9e5e0a3e1e628fe3f80b0 not found: ID does not exist" containerID="1d532ce770a46e26a9a5f71b142003624ccb9e4deea9e5e0a3e1e628fe3f80b0"
Sep 17 17:54:51.192153 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:54:51.192135    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d532ce770a46e26a9a5f71b142003624ccb9e4deea9e5e0a3e1e628fe3f80b0"} err="failed to get container status \"1d532ce770a46e26a9a5f71b142003624ccb9e4deea9e5e0a3e1e628fe3f80b0\": rpc error: code = NotFound desc = could not find container \"1d532ce770a46e26a9a5f71b142003624ccb9e4deea9e5e0a3e1e628fe3f80b0\": container with ID starting with 1d532ce770a46e26a9a5f71b142003624ccb9e4deea9e5e0a3e1e628fe3f80b0 not found: ID does not exist"
Sep 17 17:54:51.192210 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:54:51.192154    2574 scope.go:117] "RemoveContainer" containerID="93bab7f54dab7c45daedeec983fb0b4bb6b098353f835c45cced24e2d009026d"
Sep 17 17:54:51.192265 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:54:51.192201    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-7d998654b-d2jqt"]
Sep 17 17:54:51.192420 ip-10-0-134-96 kubenswrapper[2574]: E0917 17:54:51.192401    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"93bab7f54dab7c45daedeec983fb0b4bb6b098353f835c45cced24e2d009026d\": container with ID starting with 93bab7f54dab7c45daedeec983fb0b4bb6b098353f835c45cced24e2d009026d not found: ID does not exist" containerID="93bab7f54dab7c45daedeec983fb0b4bb6b098353f835c45cced24e2d009026d"
Sep 17 17:54:51.192478 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:54:51.192429    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93bab7f54dab7c45daedeec983fb0b4bb6b098353f835c45cced24e2d009026d"} err="failed to get container status \"93bab7f54dab7c45daedeec983fb0b4bb6b098353f835c45cced24e2d009026d\": rpc error: code = NotFound desc = could not find container \"93bab7f54dab7c45daedeec983fb0b4bb6b098353f835c45cced24e2d009026d\": container with ID starting with 93bab7f54dab7c45daedeec983fb0b4bb6b098353f835c45cced24e2d009026d not found: ID does not exist"
Sep 17 17:54:51.197031 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:54:51.197008    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-7d998654b-d2jqt"]
Sep 17 17:54:51.874790 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:54:51.874757    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j4ds_d3f553e8-b416-4d7d-bbef-991a0404b287/konnectivity-agent/0.log" limit=1024
Sep 17 17:54:51.935506 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:54:51.935481    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-5fb5f89d9d-qnbgw"]
Sep 17 17:54:51.935753 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:54:51.935742    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8c6366fb-8881-4eb3-b237-cd268fa59e26" containerName="kserve-container"
Sep 17 17:54:51.935795 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:54:51.935754    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="8c6366fb-8881-4eb3-b237-cd268fa59e26" containerName="kserve-container"
Sep 17 17:54:51.935795 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:54:51.935770    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8c6366fb-8881-4eb3-b237-cd268fa59e26" containerName="storage-initializer"
Sep 17 17:54:51.935795 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:54:51.935775    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="8c6366fb-8881-4eb3-b237-cd268fa59e26" containerName="storage-initializer"
Sep 17 17:54:51.935795 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:54:51.935786    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8c6366fb-8881-4eb3-b237-cd268fa59e26" containerName="kube-rbac-proxy"
Sep 17 17:54:51.935795 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:54:51.935791    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="8c6366fb-8881-4eb3-b237-cd268fa59e26" containerName="kube-rbac-proxy"
Sep 17 17:54:51.935944 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:54:51.935841    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="8c6366fb-8881-4eb3-b237-cd268fa59e26" containerName="kube-rbac-proxy"
Sep 17 17:54:51.935944 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:54:51.935850    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="8c6366fb-8881-4eb3-b237-cd268fa59e26" containerName="kserve-container"
Sep 17 17:54:51.938798 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:54:51.938781    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fb5f89d9d-qnbgw"
Sep 17 17:54:51.941376 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:54:51.941355    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 17:54:51.941494 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:54:51.941356    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-kube-rbac-proxy-sar-config\""
Sep 17 17:54:51.941593 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:54:51.941575    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 17:54:51.942256 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:54:51.942239    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 17:54:51.942337 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:54:51.942270    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 17:54:51.942337 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:54:51.942283    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-predictor-serving-cert\""
Sep 17 17:54:51.942425 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:54:51.942342    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-cm7p9\""
Sep 17 17:54:51.952968 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:54:51.952947    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-5fb5f89d9d-qnbgw"]
Sep 17 17:54:51.955990 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:54:51.955972    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b44323b3-817d-4a7e-a952-d64f03b6c6ab-cabundle-cert\") pod \"isvc-canary-promote-predictor-5fb5f89d9d-qnbgw\" (UID: \"b44323b3-817d-4a7e-a952-d64f03b6c6ab\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fb5f89d9d-qnbgw"
Sep 17 17:54:51.956124 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:54:51.956087    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b44323b3-817d-4a7e-a952-d64f03b6c6ab-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-5fb5f89d9d-qnbgw\" (UID: \"b44323b3-817d-4a7e-a952-d64f03b6c6ab\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fb5f89d9d-qnbgw"
Sep 17 17:54:51.956124 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:54:51.956117    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vjmz5\" (UniqueName: \"kubernetes.io/projected/b44323b3-817d-4a7e-a952-d64f03b6c6ab-kube-api-access-vjmz5\") pod \"isvc-canary-promote-predictor-5fb5f89d9d-qnbgw\" (UID: \"b44323b3-817d-4a7e-a952-d64f03b6c6ab\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fb5f89d9d-qnbgw"
Sep 17 17:54:51.956225 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:54:51.956142    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b44323b3-817d-4a7e-a952-d64f03b6c6ab-proxy-tls\") pod \"isvc-canary-promote-predictor-5fb5f89d9d-qnbgw\" (UID: \"b44323b3-817d-4a7e-a952-d64f03b6c6ab\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fb5f89d9d-qnbgw"
Sep 17 17:54:51.956225 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:54:51.956166    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b44323b3-817d-4a7e-a952-d64f03b6c6ab-kserve-provision-location\") pod \"isvc-canary-promote-predictor-5fb5f89d9d-qnbgw\" (UID: \"b44323b3-817d-4a7e-a952-d64f03b6c6ab\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fb5f89d9d-qnbgw"
Sep 17 17:54:51.964537 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:54:51.964519    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j4ds_d3f553e8-b416-4d7d-bbef-991a0404b287/konnectivity-agent/0.log" limit=1024
Sep 17 17:54:52.056585 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:54:52.056553    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b44323b3-817d-4a7e-a952-d64f03b6c6ab-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-5fb5f89d9d-qnbgw\" (UID: \"b44323b3-817d-4a7e-a952-d64f03b6c6ab\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fb5f89d9d-qnbgw"
Sep 17 17:54:52.056976 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:54:52.056588    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjmz5\" (UniqueName: \"kubernetes.io/projected/b44323b3-817d-4a7e-a952-d64f03b6c6ab-kube-api-access-vjmz5\") pod \"isvc-canary-promote-predictor-5fb5f89d9d-qnbgw\" (UID: \"b44323b3-817d-4a7e-a952-d64f03b6c6ab\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fb5f89d9d-qnbgw"
Sep 17 17:54:52.056976 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:54:52.056618    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b44323b3-817d-4a7e-a952-d64f03b6c6ab-proxy-tls\") pod \"isvc-canary-promote-predictor-5fb5f89d9d-qnbgw\" (UID: \"b44323b3-817d-4a7e-a952-d64f03b6c6ab\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fb5f89d9d-qnbgw"
Sep 17 17:54:52.056976 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:54:52.056638    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b44323b3-817d-4a7e-a952-d64f03b6c6ab-kserve-provision-location\") pod \"isvc-canary-promote-predictor-5fb5f89d9d-qnbgw\" (UID: \"b44323b3-817d-4a7e-a952-d64f03b6c6ab\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fb5f89d9d-qnbgw"
Sep 17 17:54:52.056976 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:54:52.056719    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b44323b3-817d-4a7e-a952-d64f03b6c6ab-cabundle-cert\") pod \"isvc-canary-promote-predictor-5fb5f89d9d-qnbgw\" (UID: \"b44323b3-817d-4a7e-a952-d64f03b6c6ab\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fb5f89d9d-qnbgw"
Sep 17 17:54:52.057230 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:54:52.057030    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j4ds_d3f553e8-b416-4d7d-bbef-991a0404b287/konnectivity-agent/0.log" limit=1024
Sep 17 17:54:52.057230 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:54:52.057173    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b44323b3-817d-4a7e-a952-d64f03b6c6ab-kserve-provision-location\") pod \"isvc-canary-promote-predictor-5fb5f89d9d-qnbgw\" (UID: \"b44323b3-817d-4a7e-a952-d64f03b6c6ab\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fb5f89d9d-qnbgw"
Sep 17 17:54:52.057318 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:54:52.057301    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b44323b3-817d-4a7e-a952-d64f03b6c6ab-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-5fb5f89d9d-qnbgw\" (UID: \"b44323b3-817d-4a7e-a952-d64f03b6c6ab\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fb5f89d9d-qnbgw"
Sep 17 17:54:52.057409 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:54:52.057393    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b44323b3-817d-4a7e-a952-d64f03b6c6ab-cabundle-cert\") pod \"isvc-canary-promote-predictor-5fb5f89d9d-qnbgw\" (UID: \"b44323b3-817d-4a7e-a952-d64f03b6c6ab\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fb5f89d9d-qnbgw"
Sep 17 17:54:52.059080 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:54:52.059064    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b44323b3-817d-4a7e-a952-d64f03b6c6ab-proxy-tls\") pod \"isvc-canary-promote-predictor-5fb5f89d9d-qnbgw\" (UID: \"b44323b3-817d-4a7e-a952-d64f03b6c6ab\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fb5f89d9d-qnbgw"
Sep 17 17:54:52.065814 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:54:52.065790    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vjmz5\" (UniqueName: \"kubernetes.io/projected/b44323b3-817d-4a7e-a952-d64f03b6c6ab-kube-api-access-vjmz5\") pod \"isvc-canary-promote-predictor-5fb5f89d9d-qnbgw\" (UID: \"b44323b3-817d-4a7e-a952-d64f03b6c6ab\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fb5f89d9d-qnbgw"
Sep 17 17:54:52.158460 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:54:52.158399    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j4ds_d3f553e8-b416-4d7d-bbef-991a0404b287/konnectivity-agent/0.log" limit=1024
Sep 17 17:54:52.248718 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:54:52.248695    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fb5f89d9d-qnbgw"
Sep 17 17:54:52.523012 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:54:52.522981    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-5fb5f89d9d-qnbgw"]
Sep 17 17:54:52.525863 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:54:52.525833    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb44323b3_817d_4a7e_a952_d64f03b6c6ab.slice/crio-3a86ba437f5d977b4e44f1121dbac8bb1b97ccae57e7fc24b66c4faebfda0039 WatchSource:0}: Error finding container 3a86ba437f5d977b4e44f1121dbac8bb1b97ccae57e7fc24b66c4faebfda0039: Status 404 returned error can't find the container with id 3a86ba437f5d977b4e44f1121dbac8bb1b97ccae57e7fc24b66c4faebfda0039
Sep 17 17:54:52.953416 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:54:52.953347    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8c6366fb-8881-4eb3-b237-cd268fa59e26" path="/var/lib/kubelet/pods/8c6366fb-8881-4eb3-b237-cd268fa59e26/volumes"
Sep 17 17:54:53.179501 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:54:53.179466    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fb5f89d9d-qnbgw" event={"ID":"b44323b3-817d-4a7e-a952-d64f03b6c6ab","Type":"ContainerStarted","Data":"96317ed58d42dd479ae1b2dfbc5009f4e18b4540b3dfe15087a9927979b7903e"}
Sep 17 17:54:53.179866 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:54:53.179524    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fb5f89d9d-qnbgw" event={"ID":"b44323b3-817d-4a7e-a952-d64f03b6c6ab","Type":"ContainerStarted","Data":"3a86ba437f5d977b4e44f1121dbac8bb1b97ccae57e7fc24b66c4faebfda0039"}
Sep 17 17:54:57.198144 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:54:57.198114    2574 generic.go:356] "Generic (PLEG): container finished" podID="b44323b3-817d-4a7e-a952-d64f03b6c6ab" containerID="96317ed58d42dd479ae1b2dfbc5009f4e18b4540b3dfe15087a9927979b7903e" exitCode=0
Sep 17 17:54:57.198524 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:54:57.198189    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fb5f89d9d-qnbgw" event={"ID":"b44323b3-817d-4a7e-a952-d64f03b6c6ab","Type":"ContainerDied","Data":"96317ed58d42dd479ae1b2dfbc5009f4e18b4540b3dfe15087a9927979b7903e"}
Sep 17 17:54:58.204060 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:54:58.204017    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fb5f89d9d-qnbgw" event={"ID":"b44323b3-817d-4a7e-a952-d64f03b6c6ab","Type":"ContainerStarted","Data":"e5374c326e7513d4546737e6e2766d6fbfedaad0c40603a3f298ffff7250fdf8"}
Sep 17 17:54:58.204470 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:54:58.204093    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fb5f89d9d-qnbgw" event={"ID":"b44323b3-817d-4a7e-a952-d64f03b6c6ab","Type":"ContainerStarted","Data":"90590e6c872a19e7d18c866d90a9356abc0de654f5445a0c494dc2aba3a6d844"}
Sep 17 17:54:58.204470 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:54:58.204399    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fb5f89d9d-qnbgw"
Sep 17 17:54:58.226379 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:54:58.226335    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fb5f89d9d-qnbgw" podStartSLOduration=3.606453887 podStartE2EDuration="7.22632087s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.619866983s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:54:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:54:58.225374039 +0000 UTC m=+1253.414247745" watchObservedRunningTime="2026-09-17 17:54:58.22632087 +0000 UTC m=+1253.415194642"
Sep 17 17:54:59.208320 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:54:59.208290    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fb5f89d9d-qnbgw"
Sep 17 17:54:59.209532 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:54:59.209503    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fb5f89d9d-qnbgw" podUID="b44323b3-817d-4a7e-a952-d64f03b6c6ab" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Sep 17 17:55:00.211915 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:55:00.211881    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fb5f89d9d-qnbgw" podUID="b44323b3-817d-4a7e-a952-d64f03b6c6ab" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Sep 17 17:55:05.216963 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:55:05.216933    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fb5f89d9d-qnbgw"
Sep 17 17:55:05.217443 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:55:05.217416    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fb5f89d9d-qnbgw" podUID="b44323b3-817d-4a7e-a952-d64f03b6c6ab" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Sep 17 17:55:15.217659 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:55:15.217617    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fb5f89d9d-qnbgw" podUID="b44323b3-817d-4a7e-a952-d64f03b6c6ab" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Sep 17 17:55:25.217777 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:55:25.217735    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fb5f89d9d-qnbgw" podUID="b44323b3-817d-4a7e-a952-d64f03b6c6ab" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Sep 17 17:55:35.217457 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:55:35.217419    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fb5f89d9d-qnbgw" podUID="b44323b3-817d-4a7e-a952-d64f03b6c6ab" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Sep 17 17:55:45.217645 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:55:45.217609    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fb5f89d9d-qnbgw" podUID="b44323b3-817d-4a7e-a952-d64f03b6c6ab" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Sep 17 17:55:55.217502 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:55:55.217461    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fb5f89d9d-qnbgw" podUID="b44323b3-817d-4a7e-a952-d64f03b6c6ab" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Sep 17 17:56:05.218915 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:56:05.218880    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fb5f89d9d-qnbgw"
Sep 17 17:56:21.993288 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:56:21.993219    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-5fb5f89d9d-qnbgw"]
Sep 17 17:56:21.993644 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:56:21.993453    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fb5f89d9d-qnbgw" podUID="b44323b3-817d-4a7e-a952-d64f03b6c6ab" containerName="kserve-container" containerID="cri-o://90590e6c872a19e7d18c866d90a9356abc0de654f5445a0c494dc2aba3a6d844" gracePeriod=30
Sep 17 17:56:21.993644 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:56:21.993489    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fb5f89d9d-qnbgw" podUID="b44323b3-817d-4a7e-a952-d64f03b6c6ab" containerName="kube-rbac-proxy" containerID="cri-o://e5374c326e7513d4546737e6e2766d6fbfedaad0c40603a3f298ffff7250fdf8" gracePeriod=30
Sep 17 17:56:22.024732 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:56:22.024702    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j4ds_d3f553e8-b416-4d7d-bbef-991a0404b287/konnectivity-agent/0.log" limit=1024
Sep 17 17:56:22.062631 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:56:22.062608    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-584d7dc949-xdnbf"]
Sep 17 17:56:22.066214 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:56:22.066200    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-584d7dc949-xdnbf"
Sep 17 17:56:22.068792 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:56:22.068775    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-v2-predictor-serving-cert\""
Sep 17 17:56:22.077181 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:56:22.077164    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-584d7dc949-xdnbf"]
Sep 17 17:56:22.133385 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:56:22.133363    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j8rsl\" (UniqueName: \"kubernetes.io/projected/fd404d77-26ef-4fc4-9bac-62d93bab0fd1-kube-api-access-j8rsl\") pod \"isvc-canary-promote-v2-predictor-584d7dc949-xdnbf\" (UID: \"fd404d77-26ef-4fc4-9bac-62d93bab0fd1\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-584d7dc949-xdnbf"
Sep 17 17:56:22.133510 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:56:22.133389    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fd404d77-26ef-4fc4-9bac-62d93bab0fd1-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-584d7dc949-xdnbf\" (UID: \"fd404d77-26ef-4fc4-9bac-62d93bab0fd1\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-584d7dc949-xdnbf"
Sep 17 17:56:22.133510 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:56:22.133408    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fd404d77-26ef-4fc4-9bac-62d93bab0fd1-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-584d7dc949-xdnbf\" (UID: \"fd404d77-26ef-4fc4-9bac-62d93bab0fd1\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-584d7dc949-xdnbf"
Sep 17 17:56:22.133510 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:56:22.133445    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fd404d77-26ef-4fc4-9bac-62d93bab0fd1-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-584d7dc949-xdnbf\" (UID: \"fd404d77-26ef-4fc4-9bac-62d93bab0fd1\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-584d7dc949-xdnbf"
Sep 17 17:56:22.133510 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:56:22.133465    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fd404d77-26ef-4fc4-9bac-62d93bab0fd1-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-584d7dc949-xdnbf\" (UID: \"fd404d77-26ef-4fc4-9bac-62d93bab0fd1\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-584d7dc949-xdnbf"
Sep 17 17:56:22.234352 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:56:22.234317    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j8rsl\" (UniqueName: \"kubernetes.io/projected/fd404d77-26ef-4fc4-9bac-62d93bab0fd1-kube-api-access-j8rsl\") pod \"isvc-canary-promote-v2-predictor-584d7dc949-xdnbf\" (UID: \"fd404d77-26ef-4fc4-9bac-62d93bab0fd1\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-584d7dc949-xdnbf"
Sep 17 17:56:22.234352 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:56:22.234348    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fd404d77-26ef-4fc4-9bac-62d93bab0fd1-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-584d7dc949-xdnbf\" (UID: \"fd404d77-26ef-4fc4-9bac-62d93bab0fd1\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-584d7dc949-xdnbf"
Sep 17 17:56:22.234569 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:56:22.234371    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fd404d77-26ef-4fc4-9bac-62d93bab0fd1-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-584d7dc949-xdnbf\" (UID: \"fd404d77-26ef-4fc4-9bac-62d93bab0fd1\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-584d7dc949-xdnbf"
Sep 17 17:56:22.234569 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:56:22.234426    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fd404d77-26ef-4fc4-9bac-62d93bab0fd1-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-584d7dc949-xdnbf\" (UID: \"fd404d77-26ef-4fc4-9bac-62d93bab0fd1\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-584d7dc949-xdnbf"
Sep 17 17:56:22.234569 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:56:22.234452    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fd404d77-26ef-4fc4-9bac-62d93bab0fd1-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-584d7dc949-xdnbf\" (UID: \"fd404d77-26ef-4fc4-9bac-62d93bab0fd1\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-584d7dc949-xdnbf"
Sep 17 17:56:22.234899 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:56:22.234877    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fd404d77-26ef-4fc4-9bac-62d93bab0fd1-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-584d7dc949-xdnbf\" (UID: \"fd404d77-26ef-4fc4-9bac-62d93bab0fd1\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-584d7dc949-xdnbf"
Sep 17 17:56:22.235094 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:56:22.235076    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fd404d77-26ef-4fc4-9bac-62d93bab0fd1-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-584d7dc949-xdnbf\" (UID: \"fd404d77-26ef-4fc4-9bac-62d93bab0fd1\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-584d7dc949-xdnbf"
Sep 17 17:56:22.235150 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:56:22.235106    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fd404d77-26ef-4fc4-9bac-62d93bab0fd1-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-584d7dc949-xdnbf\" (UID: \"fd404d77-26ef-4fc4-9bac-62d93bab0fd1\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-584d7dc949-xdnbf"
Sep 17 17:56:22.236892 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:56:22.236869    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fd404d77-26ef-4fc4-9bac-62d93bab0fd1-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-584d7dc949-xdnbf\" (UID: \"fd404d77-26ef-4fc4-9bac-62d93bab0fd1\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-584d7dc949-xdnbf"
Sep 17 17:56:22.244683 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:56:22.244634    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j8rsl\" (UniqueName: \"kubernetes.io/projected/fd404d77-26ef-4fc4-9bac-62d93bab0fd1-kube-api-access-j8rsl\") pod \"isvc-canary-promote-v2-predictor-584d7dc949-xdnbf\" (UID: \"fd404d77-26ef-4fc4-9bac-62d93bab0fd1\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-584d7dc949-xdnbf"
Sep 17 17:56:22.375470 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:56:22.375435    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-584d7dc949-xdnbf"
Sep 17 17:56:22.522135 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:56:22.522035    2574 generic.go:356] "Generic (PLEG): container finished" podID="b44323b3-817d-4a7e-a952-d64f03b6c6ab" containerID="e5374c326e7513d4546737e6e2766d6fbfedaad0c40603a3f298ffff7250fdf8" exitCode=2
Sep 17 17:56:22.522135 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:56:22.522106    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fb5f89d9d-qnbgw" event={"ID":"b44323b3-817d-4a7e-a952-d64f03b6c6ab","Type":"ContainerDied","Data":"e5374c326e7513d4546737e6e2766d6fbfedaad0c40603a3f298ffff7250fdf8"}
Sep 17 17:56:22.647579 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:56:22.647555    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-584d7dc949-xdnbf"]
Sep 17 17:56:22.650283 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:56:22.650253    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd404d77_26ef_4fc4_9bac_62d93bab0fd1.slice/crio-1054acfd10180e42ab30301be9112ea10d091a60fedafb33263bd8e8621521f3 WatchSource:0}: Error finding container 1054acfd10180e42ab30301be9112ea10d091a60fedafb33263bd8e8621521f3: Status 404 returned error can't find the container with id 1054acfd10180e42ab30301be9112ea10d091a60fedafb33263bd8e8621521f3
Sep 17 17:56:23.527613 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:56:23.527578    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-584d7dc949-xdnbf" event={"ID":"fd404d77-26ef-4fc4-9bac-62d93bab0fd1","Type":"ContainerStarted","Data":"7b845aded9ea4de8a18dbb0541f209d24a251cd6ed8e26c7a3e3a9ecdb57af6f"}
Sep 17 17:56:23.527986 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:56:23.527634    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-584d7dc949-xdnbf" event={"ID":"fd404d77-26ef-4fc4-9bac-62d93bab0fd1","Type":"ContainerStarted","Data":"1054acfd10180e42ab30301be9112ea10d091a60fedafb33263bd8e8621521f3"}
Sep 17 17:56:25.212407 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:56:25.212368    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fb5f89d9d-qnbgw" podUID="b44323b3-817d-4a7e-a952-d64f03b6c6ab" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.35:8643/healthz\": dial tcp 10.134.0.35:8643: connect: connection refused"
Sep 17 17:56:25.218175 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:56:25.218143    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fb5f89d9d-qnbgw" podUID="b44323b3-817d-4a7e-a952-d64f03b6c6ab" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Sep 17 17:56:26.532451 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:56:26.532427    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fb5f89d9d-qnbgw"
Sep 17 17:56:26.540118 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:56:26.540083    2574 generic.go:356] "Generic (PLEG): container finished" podID="b44323b3-817d-4a7e-a952-d64f03b6c6ab" containerID="90590e6c872a19e7d18c866d90a9356abc0de654f5445a0c494dc2aba3a6d844" exitCode=0
Sep 17 17:56:26.540226 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:56:26.540133    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fb5f89d9d-qnbgw"
Sep 17 17:56:26.540226 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:56:26.540140    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fb5f89d9d-qnbgw" event={"ID":"b44323b3-817d-4a7e-a952-d64f03b6c6ab","Type":"ContainerDied","Data":"90590e6c872a19e7d18c866d90a9356abc0de654f5445a0c494dc2aba3a6d844"}
Sep 17 17:56:26.540297 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:56:26.540235    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fb5f89d9d-qnbgw" event={"ID":"b44323b3-817d-4a7e-a952-d64f03b6c6ab","Type":"ContainerDied","Data":"3a86ba437f5d977b4e44f1121dbac8bb1b97ccae57e7fc24b66c4faebfda0039"}
Sep 17 17:56:26.540297 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:56:26.540275    2574 scope.go:117] "RemoveContainer" containerID="e5374c326e7513d4546737e6e2766d6fbfedaad0c40603a3f298ffff7250fdf8"
Sep 17 17:56:26.547755 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:56:26.547723    2574 scope.go:117] "RemoveContainer" containerID="90590e6c872a19e7d18c866d90a9356abc0de654f5445a0c494dc2aba3a6d844"
Sep 17 17:56:26.555873 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:56:26.555849    2574 scope.go:117] "RemoveContainer" containerID="96317ed58d42dd479ae1b2dfbc5009f4e18b4540b3dfe15087a9927979b7903e"
Sep 17 17:56:26.564798 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:56:26.564776    2574 scope.go:117] "RemoveContainer" containerID="e5374c326e7513d4546737e6e2766d6fbfedaad0c40603a3f298ffff7250fdf8"
Sep 17 17:56:26.565105 ip-10-0-134-96 kubenswrapper[2574]: E0917 17:56:26.565085    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e5374c326e7513d4546737e6e2766d6fbfedaad0c40603a3f298ffff7250fdf8\": container with ID starting with e5374c326e7513d4546737e6e2766d6fbfedaad0c40603a3f298ffff7250fdf8 not found: ID does not exist" containerID="e5374c326e7513d4546737e6e2766d6fbfedaad0c40603a3f298ffff7250fdf8"
Sep 17 17:56:26.565206 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:56:26.565119    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e5374c326e7513d4546737e6e2766d6fbfedaad0c40603a3f298ffff7250fdf8"} err="failed to get container status \"e5374c326e7513d4546737e6e2766d6fbfedaad0c40603a3f298ffff7250fdf8\": rpc error: code = NotFound desc = could not find container \"e5374c326e7513d4546737e6e2766d6fbfedaad0c40603a3f298ffff7250fdf8\": container with ID starting with e5374c326e7513d4546737e6e2766d6fbfedaad0c40603a3f298ffff7250fdf8 not found: ID does not exist"
Sep 17 17:56:26.565206 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:56:26.565145    2574 scope.go:117] "RemoveContainer" containerID="90590e6c872a19e7d18c866d90a9356abc0de654f5445a0c494dc2aba3a6d844"
Sep 17 17:56:26.565426 ip-10-0-134-96 kubenswrapper[2574]: E0917 17:56:26.565407    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"90590e6c872a19e7d18c866d90a9356abc0de654f5445a0c494dc2aba3a6d844\": container with ID starting with 90590e6c872a19e7d18c866d90a9356abc0de654f5445a0c494dc2aba3a6d844 not found: ID does not exist" containerID="90590e6c872a19e7d18c866d90a9356abc0de654f5445a0c494dc2aba3a6d844"
Sep 17 17:56:26.565465 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:56:26.565432    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"90590e6c872a19e7d18c866d90a9356abc0de654f5445a0c494dc2aba3a6d844"} err="failed to get container status \"90590e6c872a19e7d18c866d90a9356abc0de654f5445a0c494dc2aba3a6d844\": rpc error: code = NotFound desc = could not find container \"90590e6c872a19e7d18c866d90a9356abc0de654f5445a0c494dc2aba3a6d844\": container with ID starting with 90590e6c872a19e7d18c866d90a9356abc0de654f5445a0c494dc2aba3a6d844 not found: ID does not exist"
Sep 17 17:56:26.565465 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:56:26.565448    2574 scope.go:117] "RemoveContainer" containerID="96317ed58d42dd479ae1b2dfbc5009f4e18b4540b3dfe15087a9927979b7903e"
Sep 17 17:56:26.565733 ip-10-0-134-96 kubenswrapper[2574]: E0917 17:56:26.565709    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"96317ed58d42dd479ae1b2dfbc5009f4e18b4540b3dfe15087a9927979b7903e\": container with ID starting with 96317ed58d42dd479ae1b2dfbc5009f4e18b4540b3dfe15087a9927979b7903e not found: ID does not exist" containerID="96317ed58d42dd479ae1b2dfbc5009f4e18b4540b3dfe15087a9927979b7903e"
Sep 17 17:56:26.565804 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:56:26.565742    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96317ed58d42dd479ae1b2dfbc5009f4e18b4540b3dfe15087a9927979b7903e"} err="failed to get container status \"96317ed58d42dd479ae1b2dfbc5009f4e18b4540b3dfe15087a9927979b7903e\": rpc error: code = NotFound desc = could not find container \"96317ed58d42dd479ae1b2dfbc5009f4e18b4540b3dfe15087a9927979b7903e\": container with ID starting with 96317ed58d42dd479ae1b2dfbc5009f4e18b4540b3dfe15087a9927979b7903e not found: ID does not exist"
Sep 17 17:56:26.568980 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:56:26.568965    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b44323b3-817d-4a7e-a952-d64f03b6c6ab-kube-rbac-proxy-sar-config\") pod \"b44323b3-817d-4a7e-a952-d64f03b6c6ab\" (UID: \"b44323b3-817d-4a7e-a952-d64f03b6c6ab\") "
Sep 17 17:56:26.569057 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:56:26.568991    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vjmz5\" (UniqueName: \"kubernetes.io/projected/b44323b3-817d-4a7e-a952-d64f03b6c6ab-kube-api-access-vjmz5\") pod \"b44323b3-817d-4a7e-a952-d64f03b6c6ab\" (UID: \"b44323b3-817d-4a7e-a952-d64f03b6c6ab\") "
Sep 17 17:56:26.569110 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:56:26.569056    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b44323b3-817d-4a7e-a952-d64f03b6c6ab-proxy-tls\") pod \"b44323b3-817d-4a7e-a952-d64f03b6c6ab\" (UID: \"b44323b3-817d-4a7e-a952-d64f03b6c6ab\") "
Sep 17 17:56:26.569160 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:56:26.569112    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b44323b3-817d-4a7e-a952-d64f03b6c6ab-cabundle-cert\") pod \"b44323b3-817d-4a7e-a952-d64f03b6c6ab\" (UID: \"b44323b3-817d-4a7e-a952-d64f03b6c6ab\") "
Sep 17 17:56:26.569200 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:56:26.569162    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b44323b3-817d-4a7e-a952-d64f03b6c6ab-kserve-provision-location\") pod \"b44323b3-817d-4a7e-a952-d64f03b6c6ab\" (UID: \"b44323b3-817d-4a7e-a952-d64f03b6c6ab\") "
Sep 17 17:56:26.569480 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:56:26.569350    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b44323b3-817d-4a7e-a952-d64f03b6c6ab-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "b44323b3-817d-4a7e-a952-d64f03b6c6ab" (UID: "b44323b3-817d-4a7e-a952-d64f03b6c6ab"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:56:26.569574 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:56:26.569518    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b44323b3-817d-4a7e-a952-d64f03b6c6ab-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "b44323b3-817d-4a7e-a952-d64f03b6c6ab" (UID: "b44323b3-817d-4a7e-a952-d64f03b6c6ab"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:56:26.569574 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:56:26.569530    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b44323b3-817d-4a7e-a952-d64f03b6c6ab-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "b44323b3-817d-4a7e-a952-d64f03b6c6ab" (UID: "b44323b3-817d-4a7e-a952-d64f03b6c6ab"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:56:26.571397 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:56:26.571363    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b44323b3-817d-4a7e-a952-d64f03b6c6ab-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "b44323b3-817d-4a7e-a952-d64f03b6c6ab" (UID: "b44323b3-817d-4a7e-a952-d64f03b6c6ab"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:56:26.571495 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:56:26.571476    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b44323b3-817d-4a7e-a952-d64f03b6c6ab-kube-api-access-vjmz5" (OuterVolumeSpecName: "kube-api-access-vjmz5") pod "b44323b3-817d-4a7e-a952-d64f03b6c6ab" (UID: "b44323b3-817d-4a7e-a952-d64f03b6c6ab"). InnerVolumeSpecName "kube-api-access-vjmz5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:56:26.670480 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:56:26.670452    2574 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b44323b3-817d-4a7e-a952-d64f03b6c6ab-proxy-tls\") on node \"ip-10-0-134-96.ec2.internal\" DevicePath \"\""
Sep 17 17:56:26.670480 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:56:26.670480    2574 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b44323b3-817d-4a7e-a952-d64f03b6c6ab-cabundle-cert\") on node \"ip-10-0-134-96.ec2.internal\" DevicePath \"\""
Sep 17 17:56:26.670619 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:56:26.670490    2574 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b44323b3-817d-4a7e-a952-d64f03b6c6ab-kserve-provision-location\") on node \"ip-10-0-134-96.ec2.internal\" DevicePath \"\""
Sep 17 17:56:26.670619 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:56:26.670500    2574 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b44323b3-817d-4a7e-a952-d64f03b6c6ab-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-96.ec2.internal\" DevicePath \"\""
Sep 17 17:56:26.670619 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:56:26.670509    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vjmz5\" (UniqueName: \"kubernetes.io/projected/b44323b3-817d-4a7e-a952-d64f03b6c6ab-kube-api-access-vjmz5\") on node \"ip-10-0-134-96.ec2.internal\" DevicePath \"\""
Sep 17 17:56:26.867812 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:56:26.867780    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-5fb5f89d9d-qnbgw"]
Sep 17 17:56:26.874180 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:56:26.874149    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-5fb5f89d9d-qnbgw"]
Sep 17 17:56:26.953287 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:56:26.953211    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b44323b3-817d-4a7e-a952-d64f03b6c6ab" path="/var/lib/kubelet/pods/b44323b3-817d-4a7e-a952-d64f03b6c6ab/volumes"
Sep 17 17:56:27.544907 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:56:27.544874    2574 generic.go:356] "Generic (PLEG): container finished" podID="fd404d77-26ef-4fc4-9bac-62d93bab0fd1" containerID="7b845aded9ea4de8a18dbb0541f209d24a251cd6ed8e26c7a3e3a9ecdb57af6f" exitCode=0
Sep 17 17:56:27.545333 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:56:27.544952    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-584d7dc949-xdnbf" event={"ID":"fd404d77-26ef-4fc4-9bac-62d93bab0fd1","Type":"ContainerDied","Data":"7b845aded9ea4de8a18dbb0541f209d24a251cd6ed8e26c7a3e3a9ecdb57af6f"}
Sep 17 17:56:28.551309 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:56:28.551275    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-584d7dc949-xdnbf" event={"ID":"fd404d77-26ef-4fc4-9bac-62d93bab0fd1","Type":"ContainerStarted","Data":"b2c2693e2b855c2be24f68b80284d6605f5859f9fbfe711b8d5e13c26b42608a"}
Sep 17 17:56:28.551697 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:56:28.551336    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-584d7dc949-xdnbf" event={"ID":"fd404d77-26ef-4fc4-9bac-62d93bab0fd1","Type":"ContainerStarted","Data":"b129229666571b6114e036976bea16f207dbc0d687637afd278bcec7b4704c64"}
Sep 17 17:56:28.551697 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:56:28.551603    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-584d7dc949-xdnbf"
Sep 17 17:56:28.551767 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:56:28.551732    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-584d7dc949-xdnbf"
Sep 17 17:56:28.552934 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:56:28.552910    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-584d7dc949-xdnbf" podUID="fd404d77-26ef-4fc4-9bac-62d93bab0fd1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Sep 17 17:56:28.574188 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:56:28.574131    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-584d7dc949-xdnbf" podStartSLOduration=2.634671982 podStartE2EDuration="6.574112616s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.939440634s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:56:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:56:28.572250506 +0000 UTC m=+1343.761124212" watchObservedRunningTime="2026-09-17 17:56:28.574112616 +0000 UTC m=+1343.762986321"
Sep 17 17:56:29.555631 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:56:29.555590    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-584d7dc949-xdnbf" podUID="fd404d77-26ef-4fc4-9bac-62d93bab0fd1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Sep 17 17:56:34.560312 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:56:34.560284    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-584d7dc949-xdnbf"
Sep 17 17:56:34.560877 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:56:34.560849    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-584d7dc949-xdnbf" podUID="fd404d77-26ef-4fc4-9bac-62d93bab0fd1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Sep 17 17:56:44.561123 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:56:44.561082    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-584d7dc949-xdnbf" podUID="fd404d77-26ef-4fc4-9bac-62d93bab0fd1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Sep 17 17:56:54.561501 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:56:54.561464    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-584d7dc949-xdnbf" podUID="fd404d77-26ef-4fc4-9bac-62d93bab0fd1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Sep 17 17:57:04.560732 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:57:04.560694    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-584d7dc949-xdnbf" podUID="fd404d77-26ef-4fc4-9bac-62d93bab0fd1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Sep 17 17:57:14.561227 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:57:14.561191    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-584d7dc949-xdnbf" podUID="fd404d77-26ef-4fc4-9bac-62d93bab0fd1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Sep 17 17:57:24.561592 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:57:24.561555    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-584d7dc949-xdnbf" podUID="fd404d77-26ef-4fc4-9bac-62d93bab0fd1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Sep 17 17:57:34.561229 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:57:34.561192    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-584d7dc949-xdnbf" podUID="fd404d77-26ef-4fc4-9bac-62d93bab0fd1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Sep 17 17:57:37.270368 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:57:37.270339    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j4ds_d3f553e8-b416-4d7d-bbef-991a0404b287/konnectivity-agent/0.log" limit=1024
Sep 17 17:57:44.562018 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:57:44.561988    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-584d7dc949-xdnbf"
Sep 17 17:57:52.225766 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:57:52.225683    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-584d7dc949-xdnbf"]
Sep 17 17:57:52.226257 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:57:52.225962    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-584d7dc949-xdnbf" podUID="fd404d77-26ef-4fc4-9bac-62d93bab0fd1" containerName="kserve-container" containerID="cri-o://b129229666571b6114e036976bea16f207dbc0d687637afd278bcec7b4704c64" gracePeriod=30
Sep 17 17:57:52.226257 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:57:52.226003    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-584d7dc949-xdnbf" podUID="fd404d77-26ef-4fc4-9bac-62d93bab0fd1" containerName="kube-rbac-proxy" containerID="cri-o://b2c2693e2b855c2be24f68b80284d6605f5859f9fbfe711b8d5e13c26b42608a" gracePeriod=30
Sep 17 17:57:52.277282 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:57:52.277256    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j4ds_d3f553e8-b416-4d7d-bbef-991a0404b287/konnectivity-agent/0.log" limit=1024
Sep 17 17:57:52.866748 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:57:52.866718    2574 generic.go:356] "Generic (PLEG): container finished" podID="fd404d77-26ef-4fc4-9bac-62d93bab0fd1" containerID="b2c2693e2b855c2be24f68b80284d6605f5859f9fbfe711b8d5e13c26b42608a" exitCode=2
Sep 17 17:57:52.866908 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:57:52.866794    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-584d7dc949-xdnbf" event={"ID":"fd404d77-26ef-4fc4-9bac-62d93bab0fd1","Type":"ContainerDied","Data":"b2c2693e2b855c2be24f68b80284d6605f5859f9fbfe711b8d5e13c26b42608a"}
Sep 17 17:57:54.556320 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:57:54.556282    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-584d7dc949-xdnbf" podUID="fd404d77-26ef-4fc4-9bac-62d93bab0fd1" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.36:8643/healthz\": dial tcp 10.134.0.36:8643: connect: connection refused"
Sep 17 17:57:54.561659 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:57:54.561632    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-584d7dc949-xdnbf" podUID="fd404d77-26ef-4fc4-9bac-62d93bab0fd1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Sep 17 17:57:56.460814 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:57:56.460788    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-584d7dc949-xdnbf"
Sep 17 17:57:56.579031 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:57:56.578965    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fd404d77-26ef-4fc4-9bac-62d93bab0fd1-kserve-provision-location\") pod \"fd404d77-26ef-4fc4-9bac-62d93bab0fd1\" (UID: \"fd404d77-26ef-4fc4-9bac-62d93bab0fd1\") "
Sep 17 17:57:56.579031 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:57:56.579005    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j8rsl\" (UniqueName: \"kubernetes.io/projected/fd404d77-26ef-4fc4-9bac-62d93bab0fd1-kube-api-access-j8rsl\") pod \"fd404d77-26ef-4fc4-9bac-62d93bab0fd1\" (UID: \"fd404d77-26ef-4fc4-9bac-62d93bab0fd1\") "
Sep 17 17:57:56.579257 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:57:56.579067    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fd404d77-26ef-4fc4-9bac-62d93bab0fd1-proxy-tls\") pod \"fd404d77-26ef-4fc4-9bac-62d93bab0fd1\" (UID: \"fd404d77-26ef-4fc4-9bac-62d93bab0fd1\") "
Sep 17 17:57:56.579257 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:57:56.579224    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fd404d77-26ef-4fc4-9bac-62d93bab0fd1-kube-rbac-proxy-sar-config\") pod \"fd404d77-26ef-4fc4-9bac-62d93bab0fd1\" (UID: \"fd404d77-26ef-4fc4-9bac-62d93bab0fd1\") "
Sep 17 17:57:56.579369 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:57:56.579293    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fd404d77-26ef-4fc4-9bac-62d93bab0fd1-cabundle-cert\") pod \"fd404d77-26ef-4fc4-9bac-62d93bab0fd1\" (UID: \"fd404d77-26ef-4fc4-9bac-62d93bab0fd1\") "
Sep 17 17:57:56.579369 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:57:56.579328    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd404d77-26ef-4fc4-9bac-62d93bab0fd1-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "fd404d77-26ef-4fc4-9bac-62d93bab0fd1" (UID: "fd404d77-26ef-4fc4-9bac-62d93bab0fd1"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:57:56.579534 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:57:56.579517    2574 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fd404d77-26ef-4fc4-9bac-62d93bab0fd1-kserve-provision-location\") on node \"ip-10-0-134-96.ec2.internal\" DevicePath \"\""
Sep 17 17:57:56.579601 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:57:56.579576    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd404d77-26ef-4fc4-9bac-62d93bab0fd1-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "fd404d77-26ef-4fc4-9bac-62d93bab0fd1" (UID: "fd404d77-26ef-4fc4-9bac-62d93bab0fd1"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:57:56.579675 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:57:56.579655    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd404d77-26ef-4fc4-9bac-62d93bab0fd1-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "fd404d77-26ef-4fc4-9bac-62d93bab0fd1" (UID: "fd404d77-26ef-4fc4-9bac-62d93bab0fd1"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:57:56.581070 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:57:56.581035    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd404d77-26ef-4fc4-9bac-62d93bab0fd1-kube-api-access-j8rsl" (OuterVolumeSpecName: "kube-api-access-j8rsl") pod "fd404d77-26ef-4fc4-9bac-62d93bab0fd1" (UID: "fd404d77-26ef-4fc4-9bac-62d93bab0fd1"). InnerVolumeSpecName "kube-api-access-j8rsl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:57:56.581070 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:57:56.581056    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fd404d77-26ef-4fc4-9bac-62d93bab0fd1-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "fd404d77-26ef-4fc4-9bac-62d93bab0fd1" (UID: "fd404d77-26ef-4fc4-9bac-62d93bab0fd1"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:57:56.680647 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:57:56.680618    2574 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fd404d77-26ef-4fc4-9bac-62d93bab0fd1-cabundle-cert\") on node \"ip-10-0-134-96.ec2.internal\" DevicePath \"\""
Sep 17 17:57:56.680647 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:57:56.680642    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j8rsl\" (UniqueName: \"kubernetes.io/projected/fd404d77-26ef-4fc4-9bac-62d93bab0fd1-kube-api-access-j8rsl\") on node \"ip-10-0-134-96.ec2.internal\" DevicePath \"\""
Sep 17 17:57:56.680784 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:57:56.680654    2574 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fd404d77-26ef-4fc4-9bac-62d93bab0fd1-proxy-tls\") on node \"ip-10-0-134-96.ec2.internal\" DevicePath \"\""
Sep 17 17:57:56.680784 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:57:56.680669    2574 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fd404d77-26ef-4fc4-9bac-62d93bab0fd1-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-96.ec2.internal\" DevicePath \"\""
Sep 17 17:57:56.884685 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:57:56.884615    2574 generic.go:356] "Generic (PLEG): container finished" podID="fd404d77-26ef-4fc4-9bac-62d93bab0fd1" containerID="b129229666571b6114e036976bea16f207dbc0d687637afd278bcec7b4704c64" exitCode=0
Sep 17 17:57:56.884685 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:57:56.884665    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-584d7dc949-xdnbf"
Sep 17 17:57:56.884822 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:57:56.884697    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-584d7dc949-xdnbf" event={"ID":"fd404d77-26ef-4fc4-9bac-62d93bab0fd1","Type":"ContainerDied","Data":"b129229666571b6114e036976bea16f207dbc0d687637afd278bcec7b4704c64"}
Sep 17 17:57:56.884822 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:57:56.884752    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-584d7dc949-xdnbf" event={"ID":"fd404d77-26ef-4fc4-9bac-62d93bab0fd1","Type":"ContainerDied","Data":"1054acfd10180e42ab30301be9112ea10d091a60fedafb33263bd8e8621521f3"}
Sep 17 17:57:56.884822 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:57:56.884778    2574 scope.go:117] "RemoveContainer" containerID="b2c2693e2b855c2be24f68b80284d6605f5859f9fbfe711b8d5e13c26b42608a"
Sep 17 17:57:56.893384 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:57:56.893368    2574 scope.go:117] "RemoveContainer" containerID="b129229666571b6114e036976bea16f207dbc0d687637afd278bcec7b4704c64"
Sep 17 17:57:56.900344 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:57:56.900320    2574 scope.go:117] "RemoveContainer" containerID="7b845aded9ea4de8a18dbb0541f209d24a251cd6ed8e26c7a3e3a9ecdb57af6f"
Sep 17 17:57:56.907286 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:57:56.907274    2574 scope.go:117] "RemoveContainer" containerID="b2c2693e2b855c2be24f68b80284d6605f5859f9fbfe711b8d5e13c26b42608a"
Sep 17 17:57:56.907562 ip-10-0-134-96 kubenswrapper[2574]: E0917 17:57:56.907542    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b2c2693e2b855c2be24f68b80284d6605f5859f9fbfe711b8d5e13c26b42608a\": container with ID starting with b2c2693e2b855c2be24f68b80284d6605f5859f9fbfe711b8d5e13c26b42608a not found: ID does not exist" containerID="b2c2693e2b855c2be24f68b80284d6605f5859f9fbfe711b8d5e13c26b42608a"
Sep 17 17:57:56.907606 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:57:56.907570    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2c2693e2b855c2be24f68b80284d6605f5859f9fbfe711b8d5e13c26b42608a"} err="failed to get container status \"b2c2693e2b855c2be24f68b80284d6605f5859f9fbfe711b8d5e13c26b42608a\": rpc error: code = NotFound desc = could not find container \"b2c2693e2b855c2be24f68b80284d6605f5859f9fbfe711b8d5e13c26b42608a\": container with ID starting with b2c2693e2b855c2be24f68b80284d6605f5859f9fbfe711b8d5e13c26b42608a not found: ID does not exist"
Sep 17 17:57:56.907606 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:57:56.907588    2574 scope.go:117] "RemoveContainer" containerID="b129229666571b6114e036976bea16f207dbc0d687637afd278bcec7b4704c64"
Sep 17 17:57:56.907821 ip-10-0-134-96 kubenswrapper[2574]: E0917 17:57:56.907805    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b129229666571b6114e036976bea16f207dbc0d687637afd278bcec7b4704c64\": container with ID starting with b129229666571b6114e036976bea16f207dbc0d687637afd278bcec7b4704c64 not found: ID does not exist" containerID="b129229666571b6114e036976bea16f207dbc0d687637afd278bcec7b4704c64"
Sep 17 17:57:56.907876 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:57:56.907824    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b129229666571b6114e036976bea16f207dbc0d687637afd278bcec7b4704c64"} err="failed to get container status \"b129229666571b6114e036976bea16f207dbc0d687637afd278bcec7b4704c64\": rpc error: code = NotFound desc = could not find container \"b129229666571b6114e036976bea16f207dbc0d687637afd278bcec7b4704c64\": container with ID starting with b129229666571b6114e036976bea16f207dbc0d687637afd278bcec7b4704c64 not found: ID does not exist"
Sep 17 17:57:56.907876 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:57:56.907837    2574 scope.go:117] "RemoveContainer" containerID="7b845aded9ea4de8a18dbb0541f209d24a251cd6ed8e26c7a3e3a9ecdb57af6f"
Sep 17 17:57:56.908006 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:57:56.907988    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-584d7dc949-xdnbf"]
Sep 17 17:57:56.908068 ip-10-0-134-96 kubenswrapper[2574]: E0917 17:57:56.908037    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b845aded9ea4de8a18dbb0541f209d24a251cd6ed8e26c7a3e3a9ecdb57af6f\": container with ID starting with 7b845aded9ea4de8a18dbb0541f209d24a251cd6ed8e26c7a3e3a9ecdb57af6f not found: ID does not exist" containerID="7b845aded9ea4de8a18dbb0541f209d24a251cd6ed8e26c7a3e3a9ecdb57af6f"
Sep 17 17:57:56.908117 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:57:56.908066    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b845aded9ea4de8a18dbb0541f209d24a251cd6ed8e26c7a3e3a9ecdb57af6f"} err="failed to get container status \"7b845aded9ea4de8a18dbb0541f209d24a251cd6ed8e26c7a3e3a9ecdb57af6f\": rpc error: code = NotFound desc = could not find container \"7b845aded9ea4de8a18dbb0541f209d24a251cd6ed8e26c7a3e3a9ecdb57af6f\": container with ID starting with 7b845aded9ea4de8a18dbb0541f209d24a251cd6ed8e26c7a3e3a9ecdb57af6f not found: ID does not exist"
Sep 17 17:57:56.914001 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:57:56.913983    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-584d7dc949-xdnbf"]
Sep 17 17:57:56.953279 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:57:56.953260    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fd404d77-26ef-4fc4-9bac-62d93bab0fd1" path="/var/lib/kubelet/pods/fd404d77-26ef-4fc4-9bac-62d93bab0fd1/volumes"
Sep 17 17:58:06.560271 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:58:06.560234    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j4ds_d3f553e8-b416-4d7d-bbef-991a0404b287/konnectivity-agent/0.log" limit=1024
Sep 17 17:58:06.749261 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:58:06.749228    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j4ds_d3f553e8-b416-4d7d-bbef-991a0404b287/konnectivity-agent/0.log" limit=1024
Sep 17 17:58:16.569289 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:58:16.569255    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j4ds_d3f553e8-b416-4d7d-bbef-991a0404b287/konnectivity-agent/0.log" limit=1024
Sep 17 17:58:16.937203 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:58:16.937132    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j4ds_d3f553e8-b416-4d7d-bbef-991a0404b287/konnectivity-agent/0.log" limit=1024
Sep 17 17:59:06.911821 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:59:06.911795    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-45dp8_bd527736-9920-4c79-9069-1eb8be07f869/ovn-acl-logging/0.log"
Sep 17 17:59:06.912391 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:59:06.911851    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-45dp8_bd527736-9920-4c79-9069-1eb8be07f869/ovn-acl-logging/0.log"
Sep 17 17:59:21.551519 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:59:21.551417    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j4ds_d3f553e8-b416-4d7d-bbef-991a0404b287/konnectivity-agent/0.log" limit=1024
Sep 17 17:59:51.809361 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:59:51.809326    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j4ds_d3f553e8-b416-4d7d-bbef-991a0404b287/konnectivity-agent/0.log" limit=1024
Sep 17 17:59:51.892323 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:59:51.892240    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j4ds_d3f553e8-b416-4d7d-bbef-991a0404b287/konnectivity-agent/0.log" limit=1024
Sep 17 17:59:52.464998 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:59:52.464965    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-pmtlj/must-gather-j5kzr"]
Sep 17 17:59:52.465267 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:59:52.465232    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b44323b3-817d-4a7e-a952-d64f03b6c6ab" containerName="storage-initializer"
Sep 17 17:59:52.465267 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:59:52.465242    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="b44323b3-817d-4a7e-a952-d64f03b6c6ab" containerName="storage-initializer"
Sep 17 17:59:52.465267 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:59:52.465251    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b44323b3-817d-4a7e-a952-d64f03b6c6ab" containerName="kserve-container"
Sep 17 17:59:52.465267 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:59:52.465257    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="b44323b3-817d-4a7e-a952-d64f03b6c6ab" containerName="kserve-container"
Sep 17 17:59:52.465267 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:59:52.465266    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fd404d77-26ef-4fc4-9bac-62d93bab0fd1" containerName="storage-initializer"
Sep 17 17:59:52.465454 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:59:52.465272    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="fd404d77-26ef-4fc4-9bac-62d93bab0fd1" containerName="storage-initializer"
Sep 17 17:59:52.465454 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:59:52.465279    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fd404d77-26ef-4fc4-9bac-62d93bab0fd1" containerName="kube-rbac-proxy"
Sep 17 17:59:52.465454 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:59:52.465284    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="fd404d77-26ef-4fc4-9bac-62d93bab0fd1" containerName="kube-rbac-proxy"
Sep 17 17:59:52.465454 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:59:52.465299    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fd404d77-26ef-4fc4-9bac-62d93bab0fd1" containerName="kserve-container"
Sep 17 17:59:52.465454 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:59:52.465304    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="fd404d77-26ef-4fc4-9bac-62d93bab0fd1" containerName="kserve-container"
Sep 17 17:59:52.465454 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:59:52.465313    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b44323b3-817d-4a7e-a952-d64f03b6c6ab" containerName="kube-rbac-proxy"
Sep 17 17:59:52.465454 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:59:52.465319    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="b44323b3-817d-4a7e-a952-d64f03b6c6ab" containerName="kube-rbac-proxy"
Sep 17 17:59:52.465454 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:59:52.465363    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="fd404d77-26ef-4fc4-9bac-62d93bab0fd1" containerName="kube-rbac-proxy"
Sep 17 17:59:52.465454 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:59:52.465369    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="fd404d77-26ef-4fc4-9bac-62d93bab0fd1" containerName="kserve-container"
Sep 17 17:59:52.465454 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:59:52.465374    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="b44323b3-817d-4a7e-a952-d64f03b6c6ab" containerName="kube-rbac-proxy"
Sep 17 17:59:52.465454 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:59:52.465382    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="b44323b3-817d-4a7e-a952-d64f03b6c6ab" containerName="kserve-container"
Sep 17 17:59:52.468292 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:59:52.468272    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-pmtlj/must-gather-j5kzr"
Sep 17 17:59:52.470573 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:59:52.470544    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-pmtlj\"/\"default-dockercfg-9tr9w\""
Sep 17 17:59:52.471568 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:59:52.471553    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-pmtlj\"/\"openshift-service-ca.crt\""
Sep 17 17:59:52.471657 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:59:52.471570    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-pmtlj\"/\"kube-root-ca.crt\""
Sep 17 17:59:52.476339 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:59:52.476317    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-pmtlj/must-gather-j5kzr"]
Sep 17 17:59:52.478897 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:59:52.478878    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j4ds_d3f553e8-b416-4d7d-bbef-991a0404b287/konnectivity-agent/0.log" limit=1024
Sep 17 17:59:52.573723 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:59:52.573691    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j4ds_d3f553e8-b416-4d7d-bbef-991a0404b287/konnectivity-agent/0.log" limit=1024
Sep 17 17:59:52.624608 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:59:52.624581    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bmzrf\" (UniqueName: \"kubernetes.io/projected/eded7760-fce3-43a7-a52f-d525a6e205b2-kube-api-access-bmzrf\") pod \"must-gather-j5kzr\" (UID: \"eded7760-fce3-43a7-a52f-d525a6e205b2\") " pod="openshift-must-gather-pmtlj/must-gather-j5kzr"
Sep 17 17:59:52.624740 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:59:52.624621    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/eded7760-fce3-43a7-a52f-d525a6e205b2-must-gather-output\") pod \"must-gather-j5kzr\" (UID: \"eded7760-fce3-43a7-a52f-d525a6e205b2\") " pod="openshift-must-gather-pmtlj/must-gather-j5kzr"
Sep 17 17:59:52.725412 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:59:52.725346    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmzrf\" (UniqueName: \"kubernetes.io/projected/eded7760-fce3-43a7-a52f-d525a6e205b2-kube-api-access-bmzrf\") pod \"must-gather-j5kzr\" (UID: \"eded7760-fce3-43a7-a52f-d525a6e205b2\") " pod="openshift-must-gather-pmtlj/must-gather-j5kzr"
Sep 17 17:59:52.725412 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:59:52.725381    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/eded7760-fce3-43a7-a52f-d525a6e205b2-must-gather-output\") pod \"must-gather-j5kzr\" (UID: \"eded7760-fce3-43a7-a52f-d525a6e205b2\") " pod="openshift-must-gather-pmtlj/must-gather-j5kzr"
Sep 17 17:59:52.725694 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:59:52.725679    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/eded7760-fce3-43a7-a52f-d525a6e205b2-must-gather-output\") pod \"must-gather-j5kzr\" (UID: \"eded7760-fce3-43a7-a52f-d525a6e205b2\") " pod="openshift-must-gather-pmtlj/must-gather-j5kzr"
Sep 17 17:59:52.734538 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:59:52.734509    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bmzrf\" (UniqueName: \"kubernetes.io/projected/eded7760-fce3-43a7-a52f-d525a6e205b2-kube-api-access-bmzrf\") pod \"must-gather-j5kzr\" (UID: \"eded7760-fce3-43a7-a52f-d525a6e205b2\") " pod="openshift-must-gather-pmtlj/must-gather-j5kzr"
Sep 17 17:59:52.776978 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:59:52.776953    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-pmtlj/must-gather-j5kzr"
Sep 17 17:59:53.043376 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:59:53.043344    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-pmtlj/must-gather-j5kzr"]
Sep 17 17:59:53.046360 ip-10-0-134-96 kubenswrapper[2574]: W0917 17:59:53.046331    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podeded7760_fce3_43a7_a52f_d525a6e205b2.slice/crio-e0af4b5e66572bc67451b5c40d983b095229fdeef209560d5005659ac6d0718a WatchSource:0}: Error finding container e0af4b5e66572bc67451b5c40d983b095229fdeef209560d5005659ac6d0718a: Status 404 returned error can't find the container with id e0af4b5e66572bc67451b5c40d983b095229fdeef209560d5005659ac6d0718a
Sep 17 17:59:53.048398 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:59:53.048384    2574 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 17:59:53.310318 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:59:53.310242    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-pmtlj/must-gather-j5kzr" event={"ID":"eded7760-fce3-43a7-a52f-d525a6e205b2","Type":"ContainerStarted","Data":"e0af4b5e66572bc67451b5c40d983b095229fdeef209560d5005659ac6d0718a"}
Sep 17 17:59:59.340558 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:59:59.340510    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-pmtlj/must-gather-j5kzr" event={"ID":"eded7760-fce3-43a7-a52f-d525a6e205b2","Type":"ContainerStarted","Data":"898def24d23e1baf4289bdb7096e9b7eb0254b39cabab48cb2ccae2eb32d3806"}
Sep 17 17:59:59.340994 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:59:59.340563    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-pmtlj/must-gather-j5kzr" event={"ID":"eded7760-fce3-43a7-a52f-d525a6e205b2","Type":"ContainerStarted","Data":"e193a599f70fa3aa456ac38a9479d6daeb9f1e0fb6da0bba11de3388a6eb7653"}
Sep 17 17:59:59.360505 ip-10-0-134-96 kubenswrapper[2574]: I0917 17:59:59.360439    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-pmtlj/must-gather-j5kzr" podStartSLOduration=1.430984427 podStartE2EDuration="7.360416295s" totalImagesPullingTime="5.929431868s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:59:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:59:59.360033863 +0000 UTC m=+1554.548907571" watchObservedRunningTime="2026-09-17 17:59:59.360416295 +0000 UTC m=+1554.549290002"
Sep 17 18:00:22.438834 ip-10-0-134-96 kubenswrapper[2574]: I0917 18:00:22.438801    2574 generic.go:356] "Generic (PLEG): container finished" podID="eded7760-fce3-43a7-a52f-d525a6e205b2" containerID="e193a599f70fa3aa456ac38a9479d6daeb9f1e0fb6da0bba11de3388a6eb7653" exitCode=0
Sep 17 18:00:22.439209 ip-10-0-134-96 kubenswrapper[2574]: I0917 18:00:22.438873    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-pmtlj/must-gather-j5kzr" event={"ID":"eded7760-fce3-43a7-a52f-d525a6e205b2","Type":"ContainerDied","Data":"e193a599f70fa3aa456ac38a9479d6daeb9f1e0fb6da0bba11de3388a6eb7653"}
Sep 17 18:00:22.439209 ip-10-0-134-96 kubenswrapper[2574]: I0917 18:00:22.439191    2574 scope.go:117] "RemoveContainer" containerID="e193a599f70fa3aa456ac38a9479d6daeb9f1e0fb6da0bba11de3388a6eb7653"
Sep 17 18:00:23.345390 ip-10-0-134-96 kubenswrapper[2574]: I0917 18:00:23.345364    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-pmtlj_must-gather-j5kzr_eded7760-fce3-43a7-a52f-d525a6e205b2/gather/0.log"
Sep 17 18:00:23.808308 ip-10-0-134-96 kubenswrapper[2574]: I0917 18:00:23.808277    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j4ds_d3f553e8-b416-4d7d-bbef-991a0404b287/konnectivity-agent/0.log" limit=1024
Sep 17 18:00:23.965968 ip-10-0-134-96 kubenswrapper[2574]: I0917 18:00:23.965938    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j4ds_d3f553e8-b416-4d7d-bbef-991a0404b287/konnectivity-agent/0.log" limit=1024
Sep 17 18:00:24.064766 ip-10-0-134-96 kubenswrapper[2574]: I0917 18:00:24.064657    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j4ds_d3f553e8-b416-4d7d-bbef-991a0404b287/konnectivity-agent/0.log" limit=1024
Sep 17 18:00:26.864985 ip-10-0-134-96 kubenswrapper[2574]: I0917 18:00:26.864951    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-sv2qq_fadd68fa-727c-4581-bed0-5ab8b73b9982/global-pull-secret-syncer/0.log"
Sep 17 18:00:26.912092 ip-10-0-134-96 kubenswrapper[2574]: I0917 18:00:26.912062    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-2j4ds_d3f553e8-b416-4d7d-bbef-991a0404b287/konnectivity-agent/0.log"
Sep 17 18:00:27.083770 ip-10-0-134-96 kubenswrapper[2574]: I0917 18:00:27.083738    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-134-96.ec2.internal_ee6fb7c1eb5fa5842b2aa3f6f4ac1869/haproxy/0.log"
Sep 17 18:00:28.787091 ip-10-0-134-96 kubenswrapper[2574]: I0917 18:00:28.787057    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-must-gather-pmtlj/must-gather-j5kzr"]
Sep 17 18:00:28.787514 ip-10-0-134-96 kubenswrapper[2574]: I0917 18:00:28.787216    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-must-gather-pmtlj/must-gather-j5kzr" podUID="eded7760-fce3-43a7-a52f-d525a6e205b2" containerName="copy" containerID="cri-o://898def24d23e1baf4289bdb7096e9b7eb0254b39cabab48cb2ccae2eb32d3806" gracePeriod=2
Sep 17 18:00:28.795431 ip-10-0-134-96 kubenswrapper[2574]: I0917 18:00:28.795403    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-must-gather-pmtlj/must-gather-j5kzr"]
Sep 17 18:00:28.796686 ip-10-0-134-96 kubenswrapper[2574]: E0917 18:00:28.796658    2574 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"eded7760-fce3-43a7-a52f-d525a6e205b2\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"openshift-must-gather-pmtlj\"/\"must-gather-j5kzr\": pods \"must-gather-j5kzr\" not found" pod="openshift-must-gather-pmtlj/must-gather-j5kzr"
Sep 17 18:00:29.012473 ip-10-0-134-96 kubenswrapper[2574]: I0917 18:00:29.012455    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-pmtlj_must-gather-j5kzr_eded7760-fce3-43a7-a52f-d525a6e205b2/copy/0.log"
Sep 17 18:00:29.014021 ip-10-0-134-96 kubenswrapper[2574]: I0917 18:00:29.014007    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-pmtlj/must-gather-j5kzr"
Sep 17 18:00:29.155594 ip-10-0-134-96 kubenswrapper[2574]: I0917 18:00:29.155528    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bmzrf\" (UniqueName: \"kubernetes.io/projected/eded7760-fce3-43a7-a52f-d525a6e205b2-kube-api-access-bmzrf\") pod \"eded7760-fce3-43a7-a52f-d525a6e205b2\" (UID: \"eded7760-fce3-43a7-a52f-d525a6e205b2\") "
Sep 17 18:00:29.155594 ip-10-0-134-96 kubenswrapper[2574]: I0917 18:00:29.155590    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/eded7760-fce3-43a7-a52f-d525a6e205b2-must-gather-output\") pod \"eded7760-fce3-43a7-a52f-d525a6e205b2\" (UID: \"eded7760-fce3-43a7-a52f-d525a6e205b2\") "
Sep 17 18:00:29.157340 ip-10-0-134-96 kubenswrapper[2574]: I0917 18:00:29.157311    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eded7760-fce3-43a7-a52f-d525a6e205b2-must-gather-output" (OuterVolumeSpecName: "must-gather-output") pod "eded7760-fce3-43a7-a52f-d525a6e205b2" (UID: "eded7760-fce3-43a7-a52f-d525a6e205b2"). InnerVolumeSpecName "must-gather-output". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:00:29.157649 ip-10-0-134-96 kubenswrapper[2574]: I0917 18:00:29.157629    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eded7760-fce3-43a7-a52f-d525a6e205b2-kube-api-access-bmzrf" (OuterVolumeSpecName: "kube-api-access-bmzrf") pod "eded7760-fce3-43a7-a52f-d525a6e205b2" (UID: "eded7760-fce3-43a7-a52f-d525a6e205b2"). InnerVolumeSpecName "kube-api-access-bmzrf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:00:29.256421 ip-10-0-134-96 kubenswrapper[2574]: I0917 18:00:29.256390    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bmzrf\" (UniqueName: \"kubernetes.io/projected/eded7760-fce3-43a7-a52f-d525a6e205b2-kube-api-access-bmzrf\") on node \"ip-10-0-134-96.ec2.internal\" DevicePath \"\""
Sep 17 18:00:29.256421 ip-10-0-134-96 kubenswrapper[2574]: I0917 18:00:29.256416    2574 reconciler_common.go:299] "Volume detached for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/eded7760-fce3-43a7-a52f-d525a6e205b2-must-gather-output\") on node \"ip-10-0-134-96.ec2.internal\" DevicePath \"\""
Sep 17 18:00:29.466569 ip-10-0-134-96 kubenswrapper[2574]: I0917 18:00:29.466495    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-pmtlj_must-gather-j5kzr_eded7760-fce3-43a7-a52f-d525a6e205b2/copy/0.log"
Sep 17 18:00:29.466860 ip-10-0-134-96 kubenswrapper[2574]: I0917 18:00:29.466839    2574 generic.go:356] "Generic (PLEG): container finished" podID="eded7760-fce3-43a7-a52f-d525a6e205b2" containerID="898def24d23e1baf4289bdb7096e9b7eb0254b39cabab48cb2ccae2eb32d3806" exitCode=143
Sep 17 18:00:29.466954 ip-10-0-134-96 kubenswrapper[2574]: I0917 18:00:29.466892    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-pmtlj/must-gather-j5kzr"
Sep 17 18:00:29.466954 ip-10-0-134-96 kubenswrapper[2574]: I0917 18:00:29.466932    2574 scope.go:117] "RemoveContainer" containerID="898def24d23e1baf4289bdb7096e9b7eb0254b39cabab48cb2ccae2eb32d3806"
Sep 17 18:00:29.474490 ip-10-0-134-96 kubenswrapper[2574]: I0917 18:00:29.474473    2574 scope.go:117] "RemoveContainer" containerID="e193a599f70fa3aa456ac38a9479d6daeb9f1e0fb6da0bba11de3388a6eb7653"
Sep 17 18:00:29.486549 ip-10-0-134-96 kubenswrapper[2574]: I0917 18:00:29.486532    2574 scope.go:117] "RemoveContainer" containerID="898def24d23e1baf4289bdb7096e9b7eb0254b39cabab48cb2ccae2eb32d3806"
Sep 17 18:00:29.486768 ip-10-0-134-96 kubenswrapper[2574]: E0917 18:00:29.486754    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"898def24d23e1baf4289bdb7096e9b7eb0254b39cabab48cb2ccae2eb32d3806\": container with ID starting with 898def24d23e1baf4289bdb7096e9b7eb0254b39cabab48cb2ccae2eb32d3806 not found: ID does not exist" containerID="898def24d23e1baf4289bdb7096e9b7eb0254b39cabab48cb2ccae2eb32d3806"
Sep 17 18:00:29.486809 ip-10-0-134-96 kubenswrapper[2574]: I0917 18:00:29.486774    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"898def24d23e1baf4289bdb7096e9b7eb0254b39cabab48cb2ccae2eb32d3806"} err="failed to get container status \"898def24d23e1baf4289bdb7096e9b7eb0254b39cabab48cb2ccae2eb32d3806\": rpc error: code = NotFound desc = could not find container \"898def24d23e1baf4289bdb7096e9b7eb0254b39cabab48cb2ccae2eb32d3806\": container with ID starting with 898def24d23e1baf4289bdb7096e9b7eb0254b39cabab48cb2ccae2eb32d3806 not found: ID does not exist"
Sep 17 18:00:29.486809 ip-10-0-134-96 kubenswrapper[2574]: I0917 18:00:29.486792    2574 scope.go:117] "RemoveContainer" containerID="e193a599f70fa3aa456ac38a9479d6daeb9f1e0fb6da0bba11de3388a6eb7653"
Sep 17 18:00:29.487031 ip-10-0-134-96 kubenswrapper[2574]: E0917 18:00:29.487015    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e193a599f70fa3aa456ac38a9479d6daeb9f1e0fb6da0bba11de3388a6eb7653\": container with ID starting with e193a599f70fa3aa456ac38a9479d6daeb9f1e0fb6da0bba11de3388a6eb7653 not found: ID does not exist" containerID="e193a599f70fa3aa456ac38a9479d6daeb9f1e0fb6da0bba11de3388a6eb7653"
Sep 17 18:00:29.487084 ip-10-0-134-96 kubenswrapper[2574]: I0917 18:00:29.487035    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e193a599f70fa3aa456ac38a9479d6daeb9f1e0fb6da0bba11de3388a6eb7653"} err="failed to get container status \"e193a599f70fa3aa456ac38a9479d6daeb9f1e0fb6da0bba11de3388a6eb7653\": rpc error: code = NotFound desc = could not find container \"e193a599f70fa3aa456ac38a9479d6daeb9f1e0fb6da0bba11de3388a6eb7653\": container with ID starting with e193a599f70fa3aa456ac38a9479d6daeb9f1e0fb6da0bba11de3388a6eb7653 not found: ID does not exist"
Sep 17 18:00:29.893092 ip-10-0-134-96 kubenswrapper[2574]: I0917 18:00:29.893058    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j4ds_d3f553e8-b416-4d7d-bbef-991a0404b287/konnectivity-agent/0.log" limit=1024
Sep 17 18:00:29.995449 ip-10-0-134-96 kubenswrapper[2574]: I0917 18:00:29.995420    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2j4ds_d3f553e8-b416-4d7d-bbef-991a0404b287/konnectivity-agent/0.log" limit=1024
Sep 17 18:00:30.952845 ip-10-0-134-96 kubenswrapper[2574]: I0917 18:00:30.952810    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eded7760-fce3-43a7-a52f-d525a6e205b2" path="/var/lib/kubelet/pods/eded7760-fce3-43a7-a52f-d525a6e205b2/volumes"
Sep 17 18:00:31.064177 ip-10-0-134-96 kubenswrapper[2574]: I0917 18:00:31.064111    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-847b8bd8d7-5ct2x_b3d9c617-b694-40c2-98ce-3772a892df3c/cluster-monitoring-operator/0.log"
Sep 17 18:00:31.107135 ip-10-0-134-96 kubenswrapper[2574]: I0917 18:00:31.107108    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-84c885658c-tzmkz_1d665736-4c4e-49aa-ab27-8ab235e46063/kube-state-metrics/0.log"
Sep 17 18:00:31.144891 ip-10-0-134-96 kubenswrapper[2574]: I0917 18:00:31.144870    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-84c885658c-tzmkz_1d665736-4c4e-49aa-ab27-8ab235e46063/kube-rbac-proxy-main/0.log"
Sep 17 18:00:31.178233 ip-10-0-134-96 kubenswrapper[2574]: I0917 18:00:31.178211    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-84c885658c-tzmkz_1d665736-4c4e-49aa-ab27-8ab235e46063/kube-rbac-proxy-self/0.log"
Sep 17 18:00:31.222262 ip-10-0-134-96 kubenswrapper[2574]: I0917 18:00:31.222236    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-558488469d-rfr47_199968f8-4c75-40d8-bc3c-2652edf4c7da/metrics-server/0.log"
Sep 17 18:00:31.502226 ip-10-0-134-96 kubenswrapper[2574]: I0917 18:00:31.502198    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-x5649_70b48508-2726-4415-ba31-35246e0f6ad1/node-exporter/0.log"
Sep 17 18:00:31.538959 ip-10-0-134-96 kubenswrapper[2574]: I0917 18:00:31.538934    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-x5649_70b48508-2726-4415-ba31-35246e0f6ad1/kube-rbac-proxy/0.log"
Sep 17 18:00:31.568557 ip-10-0-134-96 kubenswrapper[2574]: I0917 18:00:31.568538    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-x5649_70b48508-2726-4415-ba31-35246e0f6ad1/init-textfile/0.log"
Sep 17 18:00:31.715630 ip-10-0-134-96 kubenswrapper[2574]: I0917 18:00:31.715609    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_de34e4c6-ea2c-4c68-b920-d39c07ac860d/prometheus/0.log"
Sep 17 18:00:31.742709 ip-10-0-134-96 kubenswrapper[2574]: I0917 18:00:31.742691    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_de34e4c6-ea2c-4c68-b920-d39c07ac860d/config-reloader/0.log"
Sep 17 18:00:31.792318 ip-10-0-134-96 kubenswrapper[2574]: I0917 18:00:31.792302    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_de34e4c6-ea2c-4c68-b920-d39c07ac860d/thanos-sidecar/0.log"
Sep 17 18:00:31.822578 ip-10-0-134-96 kubenswrapper[2574]: I0917 18:00:31.822560    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_de34e4c6-ea2c-4c68-b920-d39c07ac860d/kube-rbac-proxy-web/0.log"
Sep 17 18:00:31.850662 ip-10-0-134-96 kubenswrapper[2574]: I0917 18:00:31.850645    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_de34e4c6-ea2c-4c68-b920-d39c07ac860d/kube-rbac-proxy/0.log"
Sep 17 18:00:31.877260 ip-10-0-134-96 kubenswrapper[2574]: I0917 18:00:31.877241    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_de34e4c6-ea2c-4c68-b920-d39c07ac860d/kube-rbac-proxy-thanos/0.log"
Sep 17 18:00:31.903777 ip-10-0-134-96 kubenswrapper[2574]: I0917 18:00:31.903754    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_de34e4c6-ea2c-4c68-b920-d39c07ac860d/init-config-reloader/0.log"
Sep 17 18:00:34.578360 ip-10-0-134-96 kubenswrapper[2574]: I0917 18:00:34.578326    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-dx4r5/perf-node-gather-daemonset-pzbn9"]
Sep 17 18:00:34.578737 ip-10-0-134-96 kubenswrapper[2574]: I0917 18:00:34.578610    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eded7760-fce3-43a7-a52f-d525a6e205b2" containerName="gather"
Sep 17 18:00:34.578737 ip-10-0-134-96 kubenswrapper[2574]: I0917 18:00:34.578621    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="eded7760-fce3-43a7-a52f-d525a6e205b2" containerName="gather"
Sep 17 18:00:34.578737 ip-10-0-134-96 kubenswrapper[2574]: I0917 18:00:34.578629    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eded7760-fce3-43a7-a52f-d525a6e205b2" containerName="copy"
Sep 17 18:00:34.578737 ip-10-0-134-96 kubenswrapper[2574]: I0917 18:00:34.578634    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="eded7760-fce3-43a7-a52f-d525a6e205b2" containerName="copy"
Sep 17 18:00:34.578737 ip-10-0-134-96 kubenswrapper[2574]: I0917 18:00:34.578680    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="eded7760-fce3-43a7-a52f-d525a6e205b2" containerName="gather"
Sep 17 18:00:34.578737 ip-10-0-134-96 kubenswrapper[2574]: I0917 18:00:34.578691    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="eded7760-fce3-43a7-a52f-d525a6e205b2" containerName="copy"
Sep 17 18:00:34.586201 ip-10-0-134-96 kubenswrapper[2574]: I0917 18:00:34.586165    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-dx4r5/perf-node-gather-daemonset-pzbn9"
Sep 17 18:00:34.590140 ip-10-0-134-96 kubenswrapper[2574]: I0917 18:00:34.590118    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-dx4r5\"/\"openshift-service-ca.crt\""
Sep 17 18:00:34.590349 ip-10-0-134-96 kubenswrapper[2574]: I0917 18:00:34.590159    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-dx4r5\"/\"default-dockercfg-jd7vc\""
Sep 17 18:00:34.590349 ip-10-0-134-96 kubenswrapper[2574]: I0917 18:00:34.590166    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-dx4r5\"/\"kube-root-ca.crt\""
Sep 17 18:00:34.591969 ip-10-0-134-96 kubenswrapper[2574]: I0917 18:00:34.591947    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-dx4r5/perf-node-gather-daemonset-pzbn9"]
Sep 17 18:00:34.704870 ip-10-0-134-96 kubenswrapper[2574]: I0917 18:00:34.704838    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d78c6711-b236-468e-a0fa-eb34701bc7e1-sys\") pod \"perf-node-gather-daemonset-pzbn9\" (UID: \"d78c6711-b236-468e-a0fa-eb34701bc7e1\") " pod="openshift-must-gather-dx4r5/perf-node-gather-daemonset-pzbn9"
Sep 17 18:00:34.704870 ip-10-0-134-96 kubenswrapper[2574]: I0917 18:00:34.704871    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d78c6711-b236-468e-a0fa-eb34701bc7e1-podres\") pod \"perf-node-gather-daemonset-pzbn9\" (UID: \"d78c6711-b236-468e-a0fa-eb34701bc7e1\") " pod="openshift-must-gather-dx4r5/perf-node-gather-daemonset-pzbn9"
Sep 17 18:00:34.705110 ip-10-0-134-96 kubenswrapper[2574]: I0917 18:00:34.704887    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d78c6711-b236-468e-a0fa-eb34701bc7e1-proc\") pod \"perf-node-gather-daemonset-pzbn9\" (UID: \"d78c6711-b236-468e-a0fa-eb34701bc7e1\") " pod="openshift-must-gather-dx4r5/perf-node-gather-daemonset-pzbn9"
Sep 17 18:00:34.705110 ip-10-0-134-96 kubenswrapper[2574]: I0917 18:00:34.704912    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dqclp\" (UniqueName: \"kubernetes.io/projected/d78c6711-b236-468e-a0fa-eb34701bc7e1-kube-api-access-dqclp\") pod \"perf-node-gather-daemonset-pzbn9\" (UID: \"d78c6711-b236-468e-a0fa-eb34701bc7e1\") " pod="openshift-must-gather-dx4r5/perf-node-gather-daemonset-pzbn9"
Sep 17 18:00:34.705110 ip-10-0-134-96 kubenswrapper[2574]: I0917 18:00:34.705022    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d78c6711-b236-468e-a0fa-eb34701bc7e1-lib-modules\") pod \"perf-node-gather-daemonset-pzbn9\" (UID: \"d78c6711-b236-468e-a0fa-eb34701bc7e1\") " pod="openshift-must-gather-dx4r5/perf-node-gather-daemonset-pzbn9"
Sep 17 18:00:34.805569 ip-10-0-134-96 kubenswrapper[2574]: I0917 18:00:34.805522    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d78c6711-b236-468e-a0fa-eb34701bc7e1-lib-modules\") pod \"perf-node-gather-daemonset-pzbn9\" (UID: \"d78c6711-b236-468e-a0fa-eb34701bc7e1\") " pod="openshift-must-gather-dx4r5/perf-node-gather-daemonset-pzbn9"
Sep 17 18:00:34.805745 ip-10-0-134-96 kubenswrapper[2574]: I0917 18:00:34.805609    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d78c6711-b236-468e-a0fa-eb34701bc7e1-sys\") pod \"perf-node-gather-daemonset-pzbn9\" (UID: \"d78c6711-b236-468e-a0fa-eb34701bc7e1\") " pod="openshift-must-gather-dx4r5/perf-node-gather-daemonset-pzbn9"
Sep 17 18:00:34.805745 ip-10-0-134-96 kubenswrapper[2574]: I0917 18:00:34.805633    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d78c6711-b236-468e-a0fa-eb34701bc7e1-podres\") pod \"perf-node-gather-daemonset-pzbn9\" (UID: \"d78c6711-b236-468e-a0fa-eb34701bc7e1\") " pod="openshift-must-gather-dx4r5/perf-node-gather-daemonset-pzbn9"
Sep 17 18:00:34.805745 ip-10-0-134-96 kubenswrapper[2574]: I0917 18:00:34.805654    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d78c6711-b236-468e-a0fa-eb34701bc7e1-proc\") pod \"perf-node-gather-daemonset-pzbn9\" (UID: \"d78c6711-b236-468e-a0fa-eb34701bc7e1\") " pod="openshift-must-gather-dx4r5/perf-node-gather-daemonset-pzbn9"
Sep 17 18:00:34.805745 ip-10-0-134-96 kubenswrapper[2574]: I0917 18:00:34.805687    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dqclp\" (UniqueName: \"kubernetes.io/projected/d78c6711-b236-468e-a0fa-eb34701bc7e1-kube-api-access-dqclp\") pod \"perf-node-gather-daemonset-pzbn9\" (UID: \"d78c6711-b236-468e-a0fa-eb34701bc7e1\") " pod="openshift-must-gather-dx4r5/perf-node-gather-daemonset-pzbn9"
Sep 17 18:00:34.805745 ip-10-0-134-96 kubenswrapper[2574]: I0917 18:00:34.805722    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d78c6711-b236-468e-a0fa-eb34701bc7e1-lib-modules\") pod \"perf-node-gather-daemonset-pzbn9\" (UID: \"d78c6711-b236-468e-a0fa-eb34701bc7e1\") " pod="openshift-must-gather-dx4r5/perf-node-gather-daemonset-pzbn9"
Sep 17 18:00:34.805745 ip-10-0-134-96 kubenswrapper[2574]: I0917 18:00:34.805724    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d78c6711-b236-468e-a0fa-eb34701bc7e1-sys\") pod \"perf-node-gather-daemonset-pzbn9\" (UID: \"d78c6711-b236-468e-a0fa-eb34701bc7e1\") " pod="openshift-must-gather-dx4r5/perf-node-gather-daemonset-pzbn9"
Sep 17 18:00:34.805972 ip-10-0-134-96 kubenswrapper[2574]: I0917 18:00:34.805758    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d78c6711-b236-468e-a0fa-eb34701bc7e1-proc\") pod \"perf-node-gather-daemonset-pzbn9\" (UID: \"d78c6711-b236-468e-a0fa-eb34701bc7e1\") " pod="openshift-must-gather-dx4r5/perf-node-gather-daemonset-pzbn9"
Sep 17 18:00:34.805972 ip-10-0-134-96 kubenswrapper[2574]: I0917 18:00:34.805764    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d78c6711-b236-468e-a0fa-eb34701bc7e1-podres\") pod \"perf-node-gather-daemonset-pzbn9\" (UID: \"d78c6711-b236-468e-a0fa-eb34701bc7e1\") " pod="openshift-must-gather-dx4r5/perf-node-gather-daemonset-pzbn9"
Sep 17 18:00:34.814238 ip-10-0-134-96 kubenswrapper[2574]: I0917 18:00:34.814211    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dqclp\" (UniqueName: \"kubernetes.io/projected/d78c6711-b236-468e-a0fa-eb34701bc7e1-kube-api-access-dqclp\") pod \"perf-node-gather-daemonset-pzbn9\" (UID: \"d78c6711-b236-468e-a0fa-eb34701bc7e1\") " pod="openshift-must-gather-dx4r5/perf-node-gather-daemonset-pzbn9"
Sep 17 18:00:34.897338 ip-10-0-134-96 kubenswrapper[2574]: I0917 18:00:34.897260    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-dx4r5/perf-node-gather-daemonset-pzbn9"
Sep 17 18:00:35.165182 ip-10-0-134-96 kubenswrapper[2574]: I0917 18:00:35.165163    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-dx4r5/perf-node-gather-daemonset-pzbn9"]
Sep 17 18:00:35.167898 ip-10-0-134-96 kubenswrapper[2574]: W0917 18:00:35.167870    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podd78c6711_b236_468e_a0fa_eb34701bc7e1.slice/crio-f196bbf9e7ad0594aae89e49f2542e8edf593185ff55ea018318d3300acb51c7 WatchSource:0}: Error finding container f196bbf9e7ad0594aae89e49f2542e8edf593185ff55ea018318d3300acb51c7: Status 404 returned error can't find the container with id f196bbf9e7ad0594aae89e49f2542e8edf593185ff55ea018318d3300acb51c7
Sep 17 18:00:35.496764 ip-10-0-134-96 kubenswrapper[2574]: I0917 18:00:35.496726    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-dx4r5/perf-node-gather-daemonset-pzbn9" event={"ID":"d78c6711-b236-468e-a0fa-eb34701bc7e1","Type":"ContainerStarted","Data":"ccfc42c334d030785e155cfe2a4903c344e8beab0cb3664e60ffa3da484607f6"}
Sep 17 18:00:35.496937 ip-10-0-134-96 kubenswrapper[2574]: I0917 18:00:35.496777    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-dx4r5/perf-node-gather-daemonset-pzbn9" event={"ID":"d78c6711-b236-468e-a0fa-eb34701bc7e1","Type":"ContainerStarted","Data":"f196bbf9e7ad0594aae89e49f2542e8edf593185ff55ea018318d3300acb51c7"}
Sep 17 18:00:35.496937 ip-10-0-134-96 kubenswrapper[2574]: I0917 18:00:35.496858    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-dx4r5/perf-node-gather-daemonset-pzbn9"
Sep 17 18:00:36.223823 ip-10-0-134-96 kubenswrapper[2574]: I0917 18:00:36.223797    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-tnkgs_d4f6642f-b0fd-4434-a318-530eac3af0a0/dns/0.log"
Sep 17 18:00:36.261057 ip-10-0-134-96 kubenswrapper[2574]: I0917 18:00:36.261028    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-tnkgs_d4f6642f-b0fd-4434-a318-530eac3af0a0/kube-rbac-proxy/0.log"
Sep 17 18:00:36.296624 ip-10-0-134-96 kubenswrapper[2574]: I0917 18:00:36.296607    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-lf4tf_31495fae-ecfa-4263-a3db-702f75935118/dns-node-resolver/0.log"
Sep 17 18:00:36.820751 ip-10-0-134-96 kubenswrapper[2574]: I0917 18:00:36.820716    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-58dbcf67c5-dndpx_1e86520f-fe34-47cb-a4fc-2c01d1f9c9be/registry/0.log"
Sep 17 18:00:36.895535 ip-10-0-134-96 kubenswrapper[2574]: I0917 18:00:36.895509    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-t79st_67b5a7ce-5a80-4224-9f54-86ac40fa136b/node-ca/0.log"
Sep 17 18:00:38.023137 ip-10-0-134-96 kubenswrapper[2574]: I0917 18:00:38.023108    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-2sz2c_0f867410-d783-47d8-91aa-564f7e5cecc8/serve-healthcheck-canary/0.log"
Sep 17 18:00:38.535900 ip-10-0-134-96 kubenswrapper[2574]: I0917 18:00:38.535867    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-6f9c644dd9-d8vvv_d0d18822-7421-4563-8c4d-777f26ae6c0e/insights-operator/1.log"
Sep 17 18:00:38.535900 ip-10-0-134-96 kubenswrapper[2574]: I0917 18:00:38.535893    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-6f9c644dd9-d8vvv_d0d18822-7421-4563-8c4d-777f26ae6c0e/insights-operator/0.log"
Sep 17 18:00:38.659700 ip-10-0-134-96 kubenswrapper[2574]: I0917 18:00:38.659674    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-kpnbq_76b84a3e-8b66-432d-b9f3-4aeaf96d262f/kube-rbac-proxy/0.log"
Sep 17 18:00:38.695709 ip-10-0-134-96 kubenswrapper[2574]: I0917 18:00:38.695687    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-kpnbq_76b84a3e-8b66-432d-b9f3-4aeaf96d262f/exporter/0.log"
Sep 17 18:00:38.732014 ip-10-0-134-96 kubenswrapper[2574]: I0917 18:00:38.731993    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-kpnbq_76b84a3e-8b66-432d-b9f3-4aeaf96d262f/extractor/0.log"
Sep 17 18:00:41.443074 ip-10-0-134-96 kubenswrapper[2574]: I0917 18:00:41.443036    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-lpsq7_7a2a1fdf-59cd-44bc-a44e-3be744e6b2a4/s3-init/0.log"
Sep 17 18:00:41.462542 ip-10-0-134-96 kubenswrapper[2574]: I0917 18:00:41.462518    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-lpsq7_7a2a1fdf-59cd-44bc-a44e-3be744e6b2a4/download-hf-model/0.log"
Sep 17 18:00:41.509661 ip-10-0-134-96 kubenswrapper[2574]: I0917 18:00:41.509640    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-dx4r5/perf-node-gather-daemonset-pzbn9"
Sep 17 18:00:41.526643 ip-10-0-134-96 kubenswrapper[2574]: I0917 18:00:41.526600    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-dx4r5/perf-node-gather-daemonset-pzbn9" podStartSLOduration=7.526586154 podStartE2EDuration="7.526586154s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 18:00:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:00:35.513244017 +0000 UTC m=+1590.702117724" watchObservedRunningTime="2026-09-17 18:00:41.526586154 +0000 UTC m=+1596.715459858"
Sep 17 18:00:45.829530 ip-10-0-134-96 kubenswrapper[2574]: I0917 18:00:45.829502    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-76794cbd6c-7d5q6_38d5d165-e37b-4a2f-9c2a-67cbb3267248/migrator/0.log"
Sep 17 18:00:45.873303 ip-10-0-134-96 kubenswrapper[2574]: I0917 18:00:45.873272    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-76794cbd6c-7d5q6_38d5d165-e37b-4a2f-9c2a-67cbb3267248/graceful-termination/0.log"
Sep 17 18:00:46.207834 ip-10-0-134-96 kubenswrapper[2574]: I0917 18:00:46.207808    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-55889f7b47-ns2j6_88497892-29d0-4b92-803b-2ce9ebebf402/kube-storage-version-migrator-operator/1.log"
Sep 17 18:00:46.208527 ip-10-0-134-96 kubenswrapper[2574]: I0917 18:00:46.208510    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-55889f7b47-ns2j6_88497892-29d0-4b92-803b-2ce9ebebf402/kube-storage-version-migrator-operator/0.log"
Sep 17 18:00:47.573787 ip-10-0-134-96 kubenswrapper[2574]: I0917 18:00:47.573759    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-g8nlc_d7f1aae7-8961-4863-abff-c2259ef65ede/kube-multus-additional-cni-plugins/0.log"
Sep 17 18:00:47.603841 ip-10-0-134-96 kubenswrapper[2574]: I0917 18:00:47.603819    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-g8nlc_d7f1aae7-8961-4863-abff-c2259ef65ede/egress-router-binary-copy/0.log"
Sep 17 18:00:47.628378 ip-10-0-134-96 kubenswrapper[2574]: I0917 18:00:47.628358    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-g8nlc_d7f1aae7-8961-4863-abff-c2259ef65ede/cni-plugins/0.log"
Sep 17 18:00:47.656302 ip-10-0-134-96 kubenswrapper[2574]: I0917 18:00:47.656285    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-g8nlc_d7f1aae7-8961-4863-abff-c2259ef65ede/bond-cni-plugin/0.log"
Sep 17 18:00:47.690759 ip-10-0-134-96 kubenswrapper[2574]: I0917 18:00:47.690742    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-g8nlc_d7f1aae7-8961-4863-abff-c2259ef65ede/routeoverride-cni/0.log"
Sep 17 18:00:47.731036 ip-10-0-134-96 kubenswrapper[2574]: I0917 18:00:47.731016    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-g8nlc_d7f1aae7-8961-4863-abff-c2259ef65ede/whereabouts-cni-bincopy/0.log"
Sep 17 18:00:47.757217 ip-10-0-134-96 kubenswrapper[2574]: I0917 18:00:47.757198    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-g8nlc_d7f1aae7-8961-4863-abff-c2259ef65ede/whereabouts-cni/0.log"
Sep 17 18:00:48.029207 ip-10-0-134-96 kubenswrapper[2574]: I0917 18:00:48.029185    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-ck426_f33e242b-1311-43e7-8a5f-a97ad057236f/kube-multus/0.log"
Sep 17 18:00:48.053811 ip-10-0-134-96 kubenswrapper[2574]: I0917 18:00:48.053786    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-4tpm5_c18eb096-7089-426a-957b-3dc1a2d7e837/network-metrics-daemon/0.log"
Sep 17 18:00:48.082011 ip-10-0-134-96 kubenswrapper[2574]: I0917 18:00:48.081996    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-4tpm5_c18eb096-7089-426a-957b-3dc1a2d7e837/kube-rbac-proxy/0.log"
Sep 17 18:00:49.054741 ip-10-0-134-96 kubenswrapper[2574]: I0917 18:00:49.054719    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-45dp8_bd527736-9920-4c79-9069-1eb8be07f869/ovn-controller/0.log"
Sep 17 18:00:49.079904 ip-10-0-134-96 kubenswrapper[2574]: I0917 18:00:49.079861    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-45dp8_bd527736-9920-4c79-9069-1eb8be07f869/ovn-acl-logging/1.log"
Sep 17 18:00:49.080033 ip-10-0-134-96 kubenswrapper[2574]: I0917 18:00:49.079946    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-45dp8_bd527736-9920-4c79-9069-1eb8be07f869/ovn-acl-logging/0.log"
Sep 17 18:00:49.102274 ip-10-0-134-96 kubenswrapper[2574]: I0917 18:00:49.102257    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-45dp8_bd527736-9920-4c79-9069-1eb8be07f869/kube-rbac-proxy-node/0.log"
Sep 17 18:00:49.134445 ip-10-0-134-96 kubenswrapper[2574]: I0917 18:00:49.134425    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-45dp8_bd527736-9920-4c79-9069-1eb8be07f869/kube-rbac-proxy-ovn-metrics/0.log"
Sep 17 18:00:49.159881 ip-10-0-134-96 kubenswrapper[2574]: I0917 18:00:49.159864    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-45dp8_bd527736-9920-4c79-9069-1eb8be07f869/northd/0.log"
Sep 17 18:00:49.188157 ip-10-0-134-96 kubenswrapper[2574]: I0917 18:00:49.188138    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-45dp8_bd527736-9920-4c79-9069-1eb8be07f869/nbdb/0.log"
Sep 17 18:00:49.214100 ip-10-0-134-96 kubenswrapper[2574]: I0917 18:00:49.214081    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-45dp8_bd527736-9920-4c79-9069-1eb8be07f869/sbdb/0.log"
Sep 17 18:00:49.255942 ip-10-0-134-96 kubenswrapper[2574]: I0917 18:00:49.255920    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-45dp8_bd527736-9920-4c79-9069-1eb8be07f869/ovnkube-controller/0.log"
Sep 17 18:00:50.853153 ip-10-0-134-96 kubenswrapper[2574]: I0917 18:00:50.853124    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-source-7cbc45fddc-9jr2n_49831c33-9661-4705-a0e9-a24da9ea059a/check-endpoints/0.log"
Sep 17 18:00:50.910030 ip-10-0-134-96 kubenswrapper[2574]: I0917 18:00:50.910006    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-j9d2w_61c43e2e-f36c-401d-a10c-ff5f88fab086/network-check-target-container/0.log"
Sep 17 18:00:52.007119 ip-10-0-134-96 kubenswrapper[2574]: I0917 18:00:52.007093    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-xkj8c_12264c60-6721-4820-816f-798eefa04bd5/iptables-alerter/0.log"